Jul 16 11:49:00.449642 ip-10-0-129-163 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 16 11:49:00.449650 ip-10-0-129-163 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 16 11:49:00.449657 ip-10-0-129-163 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 16 11:49:00.449861 ip-10-0-129-163 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 16 11:49:10.539705 ip-10-0-129-163 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 16 11:49:10.539717 ip-10-0-129-163 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 811f03188c01490aa8e93c8e22d207a2 --
Jul 16 11:50:58.591317 ip-10-0-129-163 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 11:50:59.014932 ip-10-0-129-163 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 11:50:59.014932 ip-10-0-129-163 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 11:50:59.014932 ip-10-0-129-163 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 11:50:59.014932 ip-10-0-129-163 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 11:50:59.014932 ip-10-0-129-163 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 11:50:59.014932 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.945730    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 11:50:59.014932 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952087    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 11:50:59.014932 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952099    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 11:50:59.014932 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952103    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 11:50:59.014932 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952108    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 11:50:59.014932 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952112    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 11:50:59.014932 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952115    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 11:50:59.014932 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952117    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 11:50:59.014932 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952120    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 11:50:59.015806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952123    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 11:50:59.015806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952126    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 11:50:59.015806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952128    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 11:50:59.015806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952131    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 11:50:59.015806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952135    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 11:50:59.015806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952138    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 11:50:59.015806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952140    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 11:50:59.015806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952149    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 11:50:59.015806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952152    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 11:50:59.015806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952155    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 11:50:59.015806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952158    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 11:50:59.015806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952161    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 11:50:59.015806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952165    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 11:50:59.015806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952167    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 11:50:59.015806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952170    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 11:50:59.015806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952173    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 11:50:59.015806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952175    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 11:50:59.015806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952178    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 11:50:59.015806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952181    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 11:50:59.015806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952184    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 11:50:59.016432 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952187    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 11:50:59.016432 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952189    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 11:50:59.016432 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952192    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 11:50:59.016432 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952195    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 11:50:59.016432 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952199    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 11:50:59.016432 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952203    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 11:50:59.016432 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952206    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 11:50:59.016432 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952209    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 11:50:59.016432 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952212    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 11:50:59.016432 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952215    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 11:50:59.016432 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952218    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 11:50:59.016432 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952221    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 11:50:59.016432 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952224    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 11:50:59.016432 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952227    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 11:50:59.016432 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952229    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 11:50:59.016432 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952232    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 11:50:59.016432 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952235    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 11:50:59.016432 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952239    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 11:50:59.016432 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952241    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 11:50:59.017300 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952244    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 11:50:59.017300 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952247    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 11:50:59.017300 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952250    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 11:50:59.017300 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952252    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 11:50:59.017300 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952255    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 11:50:59.017300 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952257    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 11:50:59.017300 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952260    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 11:50:59.017300 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952263    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 11:50:59.017300 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952266    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 11:50:59.017300 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952269    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 11:50:59.017300 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952272    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 11:50:59.017300 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952274    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 11:50:59.017300 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952277    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 11:50:59.017300 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952280    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 11:50:59.017300 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952282    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 11:50:59.017300 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952285    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 11:50:59.017300 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952287    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 11:50:59.017300 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952290    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 11:50:59.017300 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952292    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 11:50:59.017300 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952295    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 11:50:59.017952 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952300    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 11:50:59.017952 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952303    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 11:50:59.017952 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952306    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 11:50:59.017952 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952309    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 11:50:59.017952 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952312    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 11:50:59.017952 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952315    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 11:50:59.017952 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952318    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 11:50:59.017952 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952320    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 11:50:59.017952 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952323    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 11:50:59.017952 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952326    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 11:50:59.017952 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952328    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 11:50:59.017952 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952331    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 11:50:59.017952 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952334    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 11:50:59.017952 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952336    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 11:50:59.017952 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952344    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 11:50:59.017952 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952347    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 11:50:59.017952 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952350    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 11:50:59.017952 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952352    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 11:50:59.017952 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952356    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 11:50:59.017952 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952360    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 11:50:59.018592 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952363    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 11:50:59.018592 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952365    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 11:50:59.018592 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952368    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 11:50:59.018592 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952370    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 11:50:59.018592 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952373    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 11:50:59.018592 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952376    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 11:50:59.018592 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952378    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 11:50:59.018592 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952381    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 11:50:59.018592 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952384    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 11:50:59.018592 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952386    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 11:50:59.018592 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952389    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 11:50:59.018592 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.952391    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 11:50:59.018592 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955079    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 11:50:59.018592 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955092    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 11:50:59.018592 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955095    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 11:50:59.018592 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955099    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 11:50:59.018592 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955102    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 11:50:59.018592 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955105    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 11:50:59.018592 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955108    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 11:50:59.018592 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955111    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 11:50:59.019239 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955114    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 11:50:59.019239 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955116    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 11:50:59.019239 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955119    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 11:50:59.019239 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955122    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 11:50:59.019239 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955125    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 11:50:59.019239 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955127    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 11:50:59.019239 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955130    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 11:50:59.019239 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955133    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 11:50:59.019239 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955135    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 11:50:59.019239 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955138    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 11:50:59.019239 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955140    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 11:50:59.019239 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955144    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 11:50:59.019239 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955146    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 11:50:59.019239 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955149    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 11:50:59.019239 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955151    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 11:50:59.019239 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955154    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 11:50:59.019239 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955157    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 11:50:59.019239 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955161    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 11:50:59.019239 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955164    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 11:50:59.019239 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955167    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 11:50:59.019872 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955169    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 11:50:59.019872 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955172    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 11:50:59.019872 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955174    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 11:50:59.019872 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955178    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 11:50:59.019872 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955181    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 11:50:59.019872 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955185    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 11:50:59.019872 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955190    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 11:50:59.019872 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955193    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 11:50:59.019872 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955197    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 11:50:59.019872 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955200    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 11:50:59.019872 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955204    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 11:50:59.019872 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955208    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 11:50:59.019872 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955211    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 11:50:59.019872 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955214    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 11:50:59.019872 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955216    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 11:50:59.019872 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955219    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 11:50:59.019872 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955222    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 11:50:59.019872 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955224    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 11:50:59.019872 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955227    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 11:50:59.020503 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955230    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 11:50:59.020503 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955232    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 11:50:59.020503 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955235    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 11:50:59.020503 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955239    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 11:50:59.020503 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955242    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 11:50:59.020503 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955244    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 11:50:59.020503 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955247    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 11:50:59.020503 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955249    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 11:50:59.020503 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955252    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 11:50:59.020503 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955255    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 11:50:59.020503 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955257    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 11:50:59.020503 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955260    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 11:50:59.020503 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955263    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 11:50:59.020503 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955265    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 11:50:59.020503 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955268    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 11:50:59.020503 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955271    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 11:50:59.020503 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955276    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 11:50:59.020503 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955280    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 11:50:59.020503 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955282    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 11:50:59.021350 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955285    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 11:50:59.021350 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955287    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 11:50:59.021350 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955290    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 11:50:59.021350 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955293    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 11:50:59.021350 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955295    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 11:50:59.021350 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955298    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 11:50:59.021350 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955301    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 11:50:59.021350 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955304    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 11:50:59.021350 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955306    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 11:50:59.021350 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955309    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 11:50:59.021350 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955311    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 11:50:59.021350 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955314    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 11:50:59.021350 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955316    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 11:50:59.021350 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955319    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 11:50:59.021350 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955321    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 11:50:59.021350 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955324    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 11:50:59.021350 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955327    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 11:50:59.021350 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955330    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 11:50:59.021350 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955333    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 11:50:59.021350 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955335    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 11:50:59.021982 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955338    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 11:50:59.021982 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955342    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 11:50:59.021982 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955345    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 11:50:59.021982 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955348    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 11:50:59.021982 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955350    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 11:50:59.021982 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955353    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 11:50:59.021982 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955355    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 11:50:59.021982 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955358    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 11:50:59.021982 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955361    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 11:50:59.021982 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955364    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 11:50:59.021982 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955366    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 11:50:59.021982 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955369    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 11:50:59.021982 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955372    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 11:50:59.021982 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955755    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 11:50:59.021982 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955760    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 11:50:59.021982 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955763    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 11:50:59.021982 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955767    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 11:50:59.021982 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955770    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 11:50:59.021982 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955773    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 11:50:59.021982 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955777    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 11:50:59.022624 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955779    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 11:50:59.022624 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955783    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 11:50:59.022624 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955786    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 11:50:59.022624 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955788    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 11:50:59.022624 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955791    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 11:50:59.022624 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955794    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 11:50:59.022624 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955796    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 11:50:59.022624 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955799    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 11:50:59.022624 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955801    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 11:50:59.022624 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955805    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 11:50:59.022624 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955807    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 11:50:59.022624 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955810    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 11:50:59.022624 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955813    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 11:50:59.022624 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955815    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 11:50:59.022624 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955818    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 11:50:59.022624 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955821    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 11:50:59.022624 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955823    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 11:50:59.022624 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955826    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 11:50:59.022624 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955828    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 11:50:59.022624 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955831    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 11:50:59.023259 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955834    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 11:50:59.023259 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955836    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 11:50:59.023259 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955839    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 11:50:59.023259 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955841    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 11:50:59.023259 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955844    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 11:50:59.023259 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955847    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 11:50:59.023259 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955849    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 11:50:59.023259 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955852    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 11:50:59.023259 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955855    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 11:50:59.023259 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955858    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 11:50:59.023259 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955861    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 11:50:59.023259 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955864    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 11:50:59.023259 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955866    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 11:50:59.023259 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955871    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 11:50:59.023259 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955874    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 11:50:59.023259 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955876    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 11:50:59.023259 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955881    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 11:50:59.023259 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955884    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 11:50:59.023259 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955886    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 11:50:59.023259 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955889    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 11:50:59.023901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955891    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 11:50:59.023901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955894    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 11:50:59.023901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955898    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 11:50:59.023901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955901    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 11:50:59.023901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955903    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 11:50:59.023901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955906    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 11:50:59.023901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955908    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 11:50:59.023901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955911    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 11:50:59.023901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955913    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 11:50:59.023901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955916    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 11:50:59.023901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955918    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 11:50:59.023901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955920    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 11:50:59.023901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955923    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 11:50:59.023901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955926    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 11:50:59.023901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955928    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 11:50:59.023901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955931    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 11:50:59.023901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955933    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 11:50:59.023901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955936    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 11:50:59.023901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955939    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 11:50:59.024540 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955942    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 11:50:59.024540 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955944    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 11:50:59.024540 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955947    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 11:50:59.024540 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955951    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 11:50:59.024540 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955954    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 11:50:59.024540 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955958    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 11:50:59.024540 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955962    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 11:50:59.024540 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955965    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 11:50:59.024540 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955968    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 11:50:59.024540 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955972    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 11:50:59.024540 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955975    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 11:50:59.024540 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955978    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 11:50:59.024540 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955981    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 11:50:59.024540 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955985    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 11:50:59.024540 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955988    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 11:50:59.024540 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955990    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 11:50:59.024540 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955994    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 11:50:59.024540 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955996    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 11:50:59.024540 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.955999    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956001    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956004    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956006    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956009    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956012    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956014    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956017    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956020    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956024    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956027    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956029    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956032    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956049    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956052    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956120    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956132    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956139    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956143    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956147    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956154    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 11:50:59.025177 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956158    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956163    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956167    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956170    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956173    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956177    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956180    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956183    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956186    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956189    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956192    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956196    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956200    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956203    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956206    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956209    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956213    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956216    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956219    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956222    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956226    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956229    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956232    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956235    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 11:50:59.025806 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956238    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956241    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956245    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956248    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956251    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956254    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956257    2578 flags.go:64] FLAG: --enable-server="true"
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956261    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956266    2578 flags.go:64] FLAG: --event-burst="100"
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956269    2578 flags.go:64] FLAG: --event-qps="50"
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956272    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956276    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956279    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956283    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956286    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956289    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956292    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956295    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956298    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956301    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956304    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956307    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956310    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956313    2578 flags.go:64] FLAG: --feature-gates=""
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956317    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 11:50:59.026648 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956320    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956323    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956326    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956329    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956332    2578 flags.go:64] FLAG: --help="false"
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956335    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956338    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956342    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956345    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956348    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956352    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956355    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956358    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956360    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956365    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956368    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956371    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956375    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956378    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956381    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956384    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956387    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956391    2578 flags.go:64] FLAG: --lock-file=""
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956393    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 11:50:59.027694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956396    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956399    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956405    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956409    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956412    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956415    2578 flags.go:64] FLAG: --logging-format="text"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956418    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956421    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956423    2578 flags.go:64] FLAG: --manifest-url=""
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956426    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956431    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956434    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956438    2578 flags.go:64] FLAG: --max-pods="110"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956441    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956444    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956447    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956450    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956453    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956456    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956459    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956469    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956472    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956477    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 11:50:59.030631 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956481    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956484    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956489    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956494    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956497    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956500    2578 flags.go:64] FLAG: --port="10250"
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956503    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956506    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-062df95385d66b05b"
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956509    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956513    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956516    2578 flags.go:64] FLAG: --register-node="true"
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956519    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956522    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956526    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956529    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956532    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956535    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956539    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956542    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956545    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956548    2578 flags.go:64] FLAG: --runonce="false"
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956550    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956553    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956556    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956559    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 11:50:59.031528 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956562    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956565    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956568    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956571    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956574    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956577    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956581    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956584    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956587    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956590    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956594    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956604    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956607    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956610    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956615    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956618    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956621    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956624    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956627    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956630    2578 flags.go:64] FLAG: --v="2"
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956634    2578 flags.go:64] FLAG: --version="false"
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956638    2578 flags.go:64] FLAG: --vmodule=""
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956642    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.956646    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956758    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 11:50:59.032760 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956761    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 11:50:59.033429 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956765    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 11:50:59.033429 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956768    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 11:50:59.033429 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956771    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 11:50:59.033429 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956774    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 11:50:59.033429 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956777    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 11:50:59.033429 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956780    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 11:50:59.033429 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956783    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 11:50:59.033429 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956787    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 11:50:59.033429 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956791    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 11:50:59.033429 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956794    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 11:50:59.033429 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956796    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 11:50:59.033429 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956799    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 11:50:59.033429 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956804    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 11:50:59.033429 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956806    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 11:50:59.033429 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956809    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 11:50:59.033429 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956812    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 11:50:59.033429 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956817    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 11:50:59.033429 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956821    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 11:50:59.033429 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956825    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 11:50:59.034263 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956827    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 11:50:59.034263 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956830    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 11:50:59.034263 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956833    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 11:50:59.034263 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956836    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 11:50:59.034263 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956839    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 11:50:59.034263 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956842    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 11:50:59.034263 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956844    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 11:50:59.034263 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956847    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 11:50:59.034263 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956849    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 11:50:59.034263 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956853    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 11:50:59.034263 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956855    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 11:50:59.034263 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956858    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 11:50:59.034263 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956860    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 11:50:59.034263 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956863    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 11:50:59.034263 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956866    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 11:50:59.034263 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956869    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 11:50:59.034263 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956871    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 11:50:59.034263 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956874    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 11:50:59.034263 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956876    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 11:50:59.034263 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956879    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 11:50:59.034902 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956882    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 11:50:59.034902 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956884    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 11:50:59.034902 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956887    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 11:50:59.034902 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956890    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 11:50:59.034902 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956892    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 11:50:59.034902 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956896    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 11:50:59.034902 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956899    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 11:50:59.034902 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956901    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 11:50:59.034902 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956904    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 11:50:59.034902 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956908    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 11:50:59.034902 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956911    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 11:50:59.034902 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956913    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 11:50:59.034902 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956916    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 11:50:59.034902 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956919    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 11:50:59.034902 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956921    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 11:50:59.034902 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956924    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 11:50:59.034902 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956927    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 11:50:59.034902 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956929    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 11:50:59.034902 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956932    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 11:50:59.034902 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956934    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 11:50:59.109282 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956937    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 11:50:59.109282 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956940    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 11:50:59.109282 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956943    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 11:50:59.109282 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956945    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 11:50:59.109282 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956948    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 11:50:59.109282 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956951    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 11:50:59.109282 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956953    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 11:50:59.109282 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956956    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 11:50:59.109282 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956959    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 11:50:59.109282 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956961    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 11:50:59.109282 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956964    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 11:50:59.109282 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956966    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 11:50:59.109282 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956969    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 11:50:59.109282 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956971    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 11:50:59.109282 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956974    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 11:50:59.109282 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956977    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 11:50:59.109282 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956979    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 11:50:59.109282 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956983    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 11:50:59.109282 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956986    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 11:50:59.043340 ip-10-0-129-163 systemd[1]: Started Kubernetes Kubelet.
Jul 16 11:50:59.110513 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956988    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 11:50:59.110513 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956991    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 11:50:59.110513 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956995    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 11:50:59.110513 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.956998    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 11:50:59.110513 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.957000    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 11:50:59.110513 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.957003    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 11:50:59.110513 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.957005    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 11:50:59.110513 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.957008    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 11:50:59.110513 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.957011    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 11:50:59.110513 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.957013    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 11:50:59.110513 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.957016    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 11:50:59.110513 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.957019    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 11:50:59.110513 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.957021    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 11:50:59.110513 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.957024    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 11:50:59.110513 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.957026    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 11:50:59.110513 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.957029    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 11:50:59.110513 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.957055    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 11:50:59.110513 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.957058    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 11:50:59.110513 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.957062    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 11:50:59.112311 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.957768    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 11:50:59.112311 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.964846    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 11:50:59.112311 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.964861    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 11:50:59.112311 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.964880    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 11:50:59.112311 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.964886    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 11:50:59.112311 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.964941    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 11:50:59.112311 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.964947    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 11:50:59.112311 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.964951    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 11:50:59.112311 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.964954    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 11:50:59.112311 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.964958    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 11:50:59.112311 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.964961    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 11:50:59.112311 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.964964    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 11:50:59.112311 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.964967    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 11:50:59.112311 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.964969    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 11:50:59.112921 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.964972    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 11:50:59.112921 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.964975    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 11:50:59.112921 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.964978    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 11:50:59.112921 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.964982    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 11:50:59.112921 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.964986    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 11:50:59.112921 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.964991    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 11:50:59.112921 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.964995    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 11:50:59.112921 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.964999    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 11:50:59.112921 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965001    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 11:50:59.112921 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965004    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 11:50:59.112921 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965007    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 11:50:59.112921 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965009    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 11:50:59.112921 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965012    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 11:50:59.112921 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965015    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 11:50:59.112921 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965018    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 11:50:59.112921 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965020    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 11:50:59.112921 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965023    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 11:50:59.112921 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965026    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 11:50:59.112921 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965028    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 11:50:59.112921 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965031    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 11:50:59.113950 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965049    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 11:50:59.113950 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965052    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 11:50:59.113950 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965055    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 11:50:59.113950 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965057    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 11:50:59.113950 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965060    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 11:50:59.113950 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965062    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 11:50:59.113950 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965065    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 11:50:59.113950 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965068    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 11:50:59.113950 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965071    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 11:50:59.113950 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965073    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 11:50:59.113950 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965076    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 11:50:59.113950 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965079    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 11:50:59.113950 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965082    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 11:50:59.113950 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965084    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 11:50:59.113950 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965087    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 11:50:59.113950 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965089    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 11:50:59.113950 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965092    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 11:50:59.113950 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965099    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 11:50:59.113950 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965102    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 11:50:59.113950 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965105    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 11:50:59.115410 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965107    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 11:50:59.115410 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965110    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 11:50:59.115410 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965113    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 11:50:59.115410 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965116    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 11:50:59.115410 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965119    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 11:50:59.115410 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965121    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 11:50:59.115410 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965124    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 11:50:59.115410 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965126    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 11:50:59.115410 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965129    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 11:50:59.115410 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965132    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 11:50:59.115410 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965135    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 11:50:59.115410 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965139    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 11:50:59.115410 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965142    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 11:50:59.115410 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965146    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 11:50:59.115410 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965148    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 11:50:59.115410 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965151    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 11:50:59.115410 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965154    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 11:50:59.115410 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965157    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 11:50:59.115410 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965160    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 11:50:59.115410 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965163    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 11:50:59.116055 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965166    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 11:50:59.116055 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965169    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 11:50:59.116055 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965172    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 11:50:59.116055 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965175    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 11:50:59.116055 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965178    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 11:50:59.116055 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965180    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 11:50:59.116055 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965183    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 11:50:59.116055 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965186    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 11:50:59.116055 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965188    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 11:50:59.116055 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965192    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 11:50:59.116055 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965197    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 11:50:59.116055 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965199    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 11:50:59.116055 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965202    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 11:50:59.116055 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965205    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 11:50:59.116055 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965208    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 11:50:59.116055 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965210    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 11:50:59.116055 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965213    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 11:50:59.116055 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965215    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 11:50:59.116055 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965218    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 11:50:59.117083 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965221    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 11:50:59.117083 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965223    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 11:50:59.117083 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965226    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 11:50:59.117083 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965228    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 11:50:59.117083 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965231    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 11:50:59.117083 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965234    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 11:50:59.117083 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965236    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 11:50:59.117083 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965239    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 11:50:59.117083 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965242    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 11:50:59.117083 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965244    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 11:50:59.117083 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965247    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 11:50:59.117083 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.965252    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 11:50:59.117083 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965364    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 11:50:59.117083 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965369    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 11:50:59.117901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965372    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 11:50:59.117901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965375    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 11:50:59.117901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965378    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 11:50:59.117901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965381    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 11:50:59.117901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965385    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 11:50:59.117901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965388    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 11:50:59.117901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965391    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 11:50:59.117901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965394    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 11:50:59.117901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965398    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 11:50:59.117901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965400    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 11:50:59.117901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965404    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 11:50:59.117901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965408    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 11:50:59.117901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965411    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 11:50:59.117901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965414    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 11:50:59.117901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965417    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 11:50:59.117901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965419    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 11:50:59.117901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965422    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 11:50:59.117901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965425    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 11:50:59.117901 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965427    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 11:50:59.119049 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965430    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 11:50:59.119049 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965432    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 11:50:59.119049 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965435    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 11:50:59.119049 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965438    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 11:50:59.119049 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965440    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 11:50:59.119049 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965443    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 11:50:59.119049 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965446    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 11:50:59.119049 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965449    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 11:50:59.119049 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965451    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 11:50:59.119049 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965454    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 11:50:59.119049 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965457    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 11:50:59.119049 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965459    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 11:50:59.119049 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965462    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 11:50:59.119049 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965464    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 11:50:59.119049 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965467    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 11:50:59.119049 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965469    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 11:50:59.119049 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965472    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 11:50:59.119049 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965474    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 11:50:59.119049 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965477    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 11:50:59.119049 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965479    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 11:50:59.120340 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965482    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 11:50:59.120340 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965486    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 11:50:59.120340 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965488    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 11:50:59.120340 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965492    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 11:50:59.120340 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965495    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 11:50:59.120340 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965498    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 11:50:59.120340 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965501    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 11:50:59.120340 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965504    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 11:50:59.120340 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965506    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 11:50:59.120340 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965509    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 11:50:59.120340 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965512    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 11:50:59.120340 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965514    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 11:50:59.120340 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965517    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 11:50:59.120340 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965519    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 11:50:59.120340 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965522    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 11:50:59.120340 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965525    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 11:50:59.120340 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965527    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 11:50:59.120340 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965530    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 11:50:59.120340 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965532    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 11:50:59.120340 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965535    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 11:50:59.122625 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965540    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 11:50:59.122625 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965543    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 11:50:59.122625 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965546    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 11:50:59.122625 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965549    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 11:50:59.122625 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965552    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 11:50:59.122625 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965555    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 11:50:59.122625 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965557    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 11:50:59.122625 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965560    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 11:50:59.122625 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965562    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 11:50:59.122625 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965565    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 11:50:59.122625 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965567    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 11:50:59.122625 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965570    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 11:50:59.122625 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965573    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 11:50:59.122625 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965577    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 11:50:59.122625 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965579    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 11:50:59.122625 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965583    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 11:50:59.122625 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965586    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 11:50:59.122625 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965589    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 11:50:59.122625 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965591    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 11:50:59.123443 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965594    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 11:50:59.123443 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965597    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 11:50:59.123443 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965600    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 11:50:59.123443 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965602    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 11:50:59.123443 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965605    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 11:50:59.123443 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965607    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 11:50:59.123443 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965610    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 11:50:59.123443 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965612    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 11:50:59.123443 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965615    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 11:50:59.123443 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965618    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 11:50:59.123443 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965620    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 11:50:59.123443 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965623    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 11:50:59.123443 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965625    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 11:50:59.123443 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965628    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 11:50:59.123443 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965631    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 11:50:59.123443 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965634    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 11:50:59.123443 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965636    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 11:50:59.123443 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965639    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 11:50:59.123443 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:50:58.965642    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 11:50:59.124864 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.965647    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 11:50:59.124864 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.965748    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 11:50:59.124864 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.968976    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 11:50:59.124864 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.970245    2578 server.go:1019] "Starting client certificate rotation"
Jul 16 11:50:59.124864 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.970343    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 11:50:59.124864 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.970381    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 11:50:59.124864 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.972952    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 11:50:59.124864 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.974709    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 11:50:59.124864 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.985046    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 16 11:50:59.124864 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.989477    2578 log.go:25] "Validated CRI v1 image API"
Jul 16 11:50:59.124864 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.990401    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 11:50:59.124864 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.995884    2578 fs.go:135] Filesystem UUIDs: map[5dd4bf49-0308-4b09-9332-ce3d1bfd07fb:/dev/nvme0n1p3 65b51de6-653c-4a59-9ca8-a66ad616a5aa:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 16 11:50:59.125273 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:58.995901    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 11:50:59.190760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.001570    2578 manager.go:216] Machine: {Timestamp:2026-07-16 11:50:59.000401558 +0000 UTC m=+0.301178668 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100094 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25fe9e8f10c351660da400643280ec SystemUUID:ec25fe9e-8f10-c351-660d-a400643280ec BootID:811f0318-8c01-490a-a8e9-3c8e22d207a2 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:79:7b:12:f6:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:79:7b:12:f6:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:a3:9b:9a:85:e3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 11:50:59.190760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.001682    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 11:50:59.190760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.001793    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 11:50:59.190760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.003084    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 11:50:59.190760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.004755    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 11:50:59.190760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.004797    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-163.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 11:50:59.190760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.005941    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.006000    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.006086    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.006909    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.008908    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.015355    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.015521    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.015555    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.015568    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.015586    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.015600    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.015611    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.016019    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.016055    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.035670    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.035677    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6nbxc"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.038157    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.038185    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.038337    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.038356    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.038370    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.038383    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.038399    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.038415    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.038981    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.038989    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.038997    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.039004    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.039019    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.039027    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.039065    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.039073    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.040867    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6nbxc"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.042432    2578 server.go:1268] "Started kubelet"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.042575    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.042648    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.042717    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.047366    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.051029    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-163.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 11:50:59.191188 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:50:59.051089    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-163.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:50:59.051166    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.051254    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:50:59.051016    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-163.ec2.internal.18c2c2dd838528f5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-163.ec2.internal,UID:ip-10-0-129-163.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-163.ec2.internal,},FirstTimestamp:2026-07-16 11:50:59.042240757 +0000 UTC m=+0.343017849,LastTimestamp:2026-07-16 11:50:59.042240757 +0000 UTC m=+0.343017849,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-163.ec2.internal,}"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.054828    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.054846    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:50:59.054890    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.055323    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.055526    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.055544    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.055612    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.055620    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.056839    2578 factory.go:55] Registering systemd factory
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.056856    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.057111    2578 factory.go:153] Registering CRI-O factory
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.057135    2578 factory.go:223] Registration of the crio container factory successfully
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.057197    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.057232    2578 factory.go:103] Registering Raw factory
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.057252    2578 manager.go:1196] Started watching for new ooms in manager
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.057684    2578 manager.go:349] Starting recovery of all containers
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.062702    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:50:59.066166    2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-163.ec2.internal\" not found" node="ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.068838    2578 manager.go:354] Recovery completed
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.072943    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.074797    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.074824    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.074835    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.075775    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.075787    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.075787    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.075849    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.076552    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.076584    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.076601    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.084530    2578 policy_none.go:49] "None policy: Start"
Jul 16 11:50:59.192257 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.084544    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 11:50:59.193320 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.084554    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 11:50:59.193320 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.086481    2578 policy_none.go:47] "Start"
Jul 16 11:50:59.193320 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:50:59.155252    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 16 11:50:59.193320 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.165568    2578 manager.go:337] "Starting Device Plugin manager"
Jul 16 11:50:59.193320 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:50:59.165621    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 11:50:59.193320 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.165632    2578 server.go:87] "Starting device plugin registration server"
Jul 16 11:50:59.193320 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.165817    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 11:50:59.193320 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.165829    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 11:50:59.193320 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.165969    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 11:50:59.193320 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.166054    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 11:50:59.193320 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.166063    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 11:50:59.193320 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:50:59.166516    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 11:50:59.193320 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:50:59.166563    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 16 11:50:59.274454 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.266468    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 11:50:59.274454 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.267960    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 11:50:59.274454 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.267986    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 11:50:59.274454 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.267999    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Jul 16 11:50:59.274454 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.268029    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.295857 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.276786    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.295857 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:50:59.276804    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-163.ec2.internal\": node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 16 11:50:59.295857 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.291802    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 11:50:59.295857 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.292973    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 11:50:59.295857 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.293003    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 11:50:59.295857 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.293020    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 11:50:59.295857 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:50:59.293064    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 11:50:59.295857 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.295077    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 11:50:59.309080 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:50:59.308735    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.393760    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"]
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.393822    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.395672    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.395702    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.395717    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.398374    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.398543    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.398581    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:50:59.408876    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.418899    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.418921    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.418933    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.418948    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.418968    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.418978    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.421243    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.421268    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.421997    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.422024    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.422038    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.457173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f84d25049db764d4ac62163954d83d46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal\" (UID: \"f84d25049db764d4ac62163954d83d46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.457197    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84c1f669f09517dfcf8e52b12af2bb23-config\") pod \"kube-apiserver-proxy-ip-10-0-129-163.ec2.internal\" (UID: \"84c1f669f09517dfcf8e52b12af2bb23\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.462838 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.457215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f84d25049db764d4ac62163954d83d46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal\" (UID: \"f84d25049db764d4ac62163954d83d46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.466381 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:50:59.464611    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-163.ec2.internal\" not found" node="ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.475599 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:50:59.469860    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-163.ec2.internal\" not found" node="ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.546940 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:50:59.510034    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 16 11:50:59.558202 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.557637    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84c1f669f09517dfcf8e52b12af2bb23-config\") pod \"kube-apiserver-proxy-ip-10-0-129-163.ec2.internal\" (UID: \"84c1f669f09517dfcf8e52b12af2bb23\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.558202 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.557667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f84d25049db764d4ac62163954d83d46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal\" (UID: \"f84d25049db764d4ac62163954d83d46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.558202 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.557716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f84d25049db764d4ac62163954d83d46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal\" (UID: \"f84d25049db764d4ac62163954d83d46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.558202 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.557734    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84c1f669f09517dfcf8e52b12af2bb23-config\") pod \"kube-apiserver-proxy-ip-10-0-129-163.ec2.internal\" (UID: \"84c1f669f09517dfcf8e52b12af2bb23\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.558202 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.557760    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f84d25049db764d4ac62163954d83d46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal\" (UID: \"f84d25049db764d4ac62163954d83d46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.558202 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.557782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f84d25049db764d4ac62163954d83d46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal\" (UID: \"f84d25049db764d4ac62163954d83d46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.650836 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:50:59.612067    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 16 11:50:59.742712 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:50:59.716383    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 16 11:50:59.807419 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.767710    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.807419 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.774394    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.817188 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:50:59.817164    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 16 11:50:59.920298 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:50:59.920254    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 16 11:50:59.934989 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.934968    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 11:50:59.955082 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.955058    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.968932 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.968913    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 11:50:59.970092 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.970070    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 11:50:59.970210 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.970198    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 11:50:59.970263 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.970237    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 11:50:59.970308 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.970237    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 11:50:59.970760 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.970747    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"
Jul 16 11:50:59.989369 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:50:59.989343    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 11:51:00.017354 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.017322    2578 apiserver.go:52] "Watching apiserver"
Jul 16 11:51:00.025625 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.025601    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 11:51:00.025976 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.025956    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-vlbks","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal","openshift-multus/multus-z7mwh","openshift-network-diagnostics/network-check-target-8p6dq","openshift-network-operator/iptables-alerter-cl9fk","openshift-ovn-kubernetes/ovnkube-node-kbw5x","kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz","openshift-cluster-node-tuning-operator/tuned-h8jpz","openshift-dns/node-resolver-4sj9h","openshift-multus/multus-additional-cni-plugins-lgflz","openshift-multus/network-metrics-daemon-4p7gl","kube-system/global-pull-secret-syncer-w5wfm","kube-system/konnectivity-agent-mv5xr"]
Jul 16 11:51:00.028454 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.028440    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vlbks"
Jul 16 11:51:00.030600 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.030575    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.031077 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.031050    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 11:51:00.031077 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.031053    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 11:51:00.031229 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.031114    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 11:51:00.031229 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.031189    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rjtcr\""
Jul 16 11:51:00.033039 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.032991    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:00.034001 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:00.033308    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p6dq" podUID="3768287e-bbac-495d-a015-f6fca7d1ec36"
Jul 16 11:51:00.034001 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.033659    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 11:51:00.034001 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.033711    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 11:51:00.034001 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.033919    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 11:51:00.034270 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.034065    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 11:51:00.034270 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.034201    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-478lj\""
Jul 16 11:51:00.036143 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.036117    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cl9fk"
Jul 16 11:51:00.038454 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.038418    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8mn8n\""
Jul 16 11:51:00.038454 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.038438    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 11:51:00.038667 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.038462    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 11:51:00.038667 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.038547    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.038667 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.038601    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 11:51:00.039623 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.039608    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 11:51:00.040891 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.040871    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 11:51:00.040998 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.040980    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.041103 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.041087    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 11:51:00.041153 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.041103    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 11:51:00.041153 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.041112    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 11:51:00.041254 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.041238    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 11:51:00.041626 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.041612    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 11:51:00.042212 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.042190    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vx7sq\""
Jul 16 11:51:00.043281 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.043263    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.043349 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.043282    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 11:51:00.043675 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.043660    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 11:51:00.043866 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.043853    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 11:51:00.043932 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.043873    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zkfwc\""
Jul 16 11:51:00.045513 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.045496    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 11:51:00.045513 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.045509    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 11:51:00.045759 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.045733    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p5j8l\""
Jul 16 11:51:00.045927 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.045816    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4sj9h"
Jul 16 11:51:00.047931 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.047914    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5pm9d\""
Jul 16 11:51:00.048041 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.048024    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 11:51:00.048208 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.048179    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 11:51:00.048354 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.048341    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.049433 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.049392    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 11:45:59 +0000 UTC" deadline="2028-04-21 22:18:56.35504816 +0000 UTC"
Jul 16 11:51:00.049433 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.049432    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15490h27m56.305618722s"
Jul 16 11:51:00.050859 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.050844    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:00.050953 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:00.050908    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p7gl" podUID="18353093-2e17-408e-b541-5dfb339343f0"
Jul 16 11:51:00.052314 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.052295    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 11:51:00.052524 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.052509    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 11:51:00.052637 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.052624    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-phcc5\""
Jul 16 11:51:00.053340 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.053326    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:00.053395 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:00.053379    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5wfm" podUID="32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7"
Jul 16 11:51:00.055552 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.055536    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 11:51:00.055657 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.055643    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mv5xr"
Jul 16 11:51:00.057959 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.057919    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 11:51:00.058025 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.057974    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gmzbz\""
Jul 16 11:51:00.058080 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.058055    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 11:51:00.061974 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.061950    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc1f2e2f-da1b-43e6-abd5-87ec82aee7e3-konnectivity-ca\") pod \"konnectivity-agent-mv5xr\" (UID: \"dc1f2e2f-da1b-43e6-abd5-87ec82aee7e3\") " pod="kube-system/konnectivity-agent-mv5xr"
Jul 16 11:51:00.062072 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.061985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-host-var-lib-cni-bin\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.062072 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-host-run-multus-certs\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.062548 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.062690 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb8f8012-4953-426c-8951-cb5dd0c048c1-env-overrides\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.062690 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prlkj\" (UniqueName: \"kubernetes.io/projected/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-kube-api-access-prlkj\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.062690 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-multus-socket-dir-parent\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.062690 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062604    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-host-run-k8s-cni-cncf-io\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.062690 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-run-openvswitch\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.062690 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns68k\" (UniqueName: \"kubernetes.io/projected/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-kube-api-access-ns68k\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.062690 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54m8w\" (UniqueName: \"kubernetes.io/projected/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-kube-api-access-54m8w\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.062690 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.063139 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-etc-kubernetes\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.063139 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-var-lib-openvswitch\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.063139 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062737    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-sysctl-d\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.063139 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-sysctl-conf\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.063139 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-var-lib-kubelet\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.063139 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb8f8012-4953-426c-8951-cb5dd0c048c1-ovnkube-config\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.063139 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-kubelet-config\") pod \"global-pull-secret-syncer-w5wfm\" (UID: \"32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7\") " pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:00.063139 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062870    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-device-dir\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.063139 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-run-netns\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.063139 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs\") pod \"network-metrics-daemon-4p7gl\" (UID: \"18353093-2e17-408e-b541-5dfb339343f0\") " pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:00.063139 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062923    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4lbl\" (UniqueName: \"kubernetes.io/projected/18353093-2e17-408e-b541-5dfb339343f0-kube-api-access-v4lbl\") pod \"network-metrics-daemon-4p7gl\" (UID: \"18353093-2e17-408e-b541-5dfb339343f0\") " pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:00.063139 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062938    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-hostroot\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.063139 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062953    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb8f8012-4953-426c-8951-cb5dd0c048c1-ovnkube-script-lib\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.063139 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/921209ea-e227-46b4-bb01-54ea362d80f0-host-slash\") pod \"iptables-alerter-cl9fk\" (UID: \"921209ea-e227-46b4-bb01-54ea362d80f0\") " pod="openshift-network-operator/iptables-alerter-cl9fk"
Jul 16 11:51:00.063846 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.062988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l64k2\" (UniqueName: \"kubernetes.io/projected/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-kube-api-access-l64k2\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.063846 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063005    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-cnibin\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.063846 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-host-var-lib-kubelet\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.063846 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063094    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-systemd\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.063846 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-sys-fs\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.063846 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.063846 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-system-cni-dir\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.063846 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063188    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-slash\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.063846 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-etc-openvswitch\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.063846 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063223    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-run-ovn\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.063846 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-lib-modules\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.063846 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4gmx\" (UniqueName: \"kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx\") pod \"network-check-target-8p6dq\" (UID: \"3768287e-bbac-495d-a015-f6fca7d1ec36\") " pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:00.063846 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/921209ea-e227-46b4-bb01-54ea362d80f0-iptables-alerter-script\") pod \"iptables-alerter-cl9fk\" (UID: \"921209ea-e227-46b4-bb01-54ea362d80f0\") " pod="openshift-network-operator/iptables-alerter-cl9fk"
Jul 16 11:51:00.063846 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-cni-binary-copy\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.064464 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret\") pod \"global-pull-secret-syncer-w5wfm\" (UID: \"32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7\") " pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:00.064464 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063364    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-etc-selinux\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.064464 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063379    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-host-run-netns\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.064464 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-kubernetes\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.064464 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063431    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-tmp\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.064464 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj6m4\" (UniqueName: \"kubernetes.io/projected/921209ea-e227-46b4-bb01-54ea362d80f0-kube-api-access-sj6m4\") pod \"iptables-alerter-cl9fk\" (UID: \"921209ea-e227-46b4-bb01-54ea362d80f0\") " pod="openshift-network-operator/iptables-alerter-cl9fk"
Jul 16 11:51:00.064464 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-cnibin\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.064464 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-host-var-lib-cni-multus\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.064464 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-run-systemd\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.064464 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063534    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb8f8012-4953-426c-8951-cb5dd0c048c1-ovn-node-metrics-cert\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.064464 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57wff\" (UniqueName: \"kubernetes.io/projected/eb8f8012-4953-426c-8951-cb5dd0c048c1-kube-api-access-57wff\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.064464 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063592    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-run\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.064464 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-tuned\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.064464 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5d99cb5-959d-4a75-89e7-fa68635e416d-serviceca\") pod \"node-ca-vlbks\" (UID: \"c5d99cb5-959d-4a75-89e7-fa68635e416d\") " pod="openshift-image-registry/node-ca-vlbks"
Jul 16 11:51:00.065047 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-os-release\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.065047 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063740    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-node-log\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.065047 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063775    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-cni-netd\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.065047 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-sysconfig\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.065047 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063841    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-dbus\") pod \"global-pull-secret-syncer-w5wfm\" (UID: \"32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7\") " pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:00.065047 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5d99cb5-959d-4a75-89e7-fa68635e416d-host\") pod \"node-ca-vlbks\" (UID: \"c5d99cb5-959d-4a75-89e7-fa68635e416d\") " pod="openshift-image-registry/node-ca-vlbks"
Jul 16 11:51:00.065047 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkzf5\" (UniqueName: \"kubernetes.io/projected/c5d99cb5-959d-4a75-89e7-fa68635e416d-kube-api-access-rkzf5\") pod \"node-ca-vlbks\" (UID: \"c5d99cb5-959d-4a75-89e7-fa68635e416d\") " pod="openshift-image-registry/node-ca-vlbks"
Jul 16 11:51:00.065047 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-cni-binary-copy\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.065047 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc1f2e2f-da1b-43e6-abd5-87ec82aee7e3-agent-certs\") pod \"konnectivity-agent-mv5xr\" (UID: \"dc1f2e2f-da1b-43e6-abd5-87ec82aee7e3\") " pod="kube-system/konnectivity-agent-mv5xr"
Jul 16 11:51:00.065047 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.063992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-log-socket\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.065047 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.064024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-os-release\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.065047 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.064070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.065047 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.064097    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-multus-cni-dir\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.065047 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.064126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-multus-daemon-config\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.065535 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.064152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-sys\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.065535 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.064220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-host\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.065535 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.064245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b123f16-31a0-49be-906c-017bdc2602e6-tmp-dir\") pod \"node-resolver-4sj9h\" (UID: \"8b123f16-31a0-49be-906c-017bdc2602e6\") " pod="openshift-dns/node-resolver-4sj9h"
Jul 16 11:51:00.065535 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.064269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.065535 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.064296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-registration-dir\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.065535 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.064321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-multus-conf-dir\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.065535 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.064348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-kubelet\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.065535 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.064373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-systemd-units\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.065535 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.064420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-cni-bin\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.065535 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.064446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.065535 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.064472    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-modprobe-d\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.065535 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.064514    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b123f16-31a0-49be-906c-017bdc2602e6-hosts-file\") pod \"node-resolver-4sj9h\" (UID: \"8b123f16-31a0-49be-906c-017bdc2602e6\") " pod="openshift-dns/node-resolver-4sj9h"
Jul 16 11:51:00.065535 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.064548    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-system-cni-dir\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.065535 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.064577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g952t\" (UniqueName: \"kubernetes.io/projected/8b123f16-31a0-49be-906c-017bdc2602e6-kube-api-access-g952t\") pod \"node-resolver-4sj9h\" (UID: \"8b123f16-31a0-49be-906c-017bdc2602e6\") " pod="openshift-dns/node-resolver-4sj9h"
Jul 16 11:51:00.065995 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.064630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-socket-dir\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.073519 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.073403    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 11:51:00.105922 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.105901    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rqggx"
Jul 16 11:51:00.116335 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.116314    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rqggx"
Jul 16 11:51:00.155904 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.155885    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 11:51:00.165793 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.165767    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-cni-binary-copy\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.165898 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.165796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc1f2e2f-da1b-43e6-abd5-87ec82aee7e3-agent-certs\") pod \"konnectivity-agent-mv5xr\" (UID: \"dc1f2e2f-da1b-43e6-abd5-87ec82aee7e3\") " pod="kube-system/konnectivity-agent-mv5xr"
Jul 16 11:51:00.165898 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.165813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-log-socket\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.165898 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.165829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-os-release\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.165898 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.165879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.166098 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.165908    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-multus-cni-dir\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.166098 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.165931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-multus-daemon-config\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.166098 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.165934    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-log-socket\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.166098 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.165956    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-sys\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.166098 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.165966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-os-release\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.166098 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.165982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-host\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.166098 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166005    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b123f16-31a0-49be-906c-017bdc2602e6-tmp-dir\") pod \"node-resolver-4sj9h\" (UID: \"8b123f16-31a0-49be-906c-017bdc2602e6\") " pod="openshift-dns/node-resolver-4sj9h"
Jul 16 11:51:00.166098 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.166098 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-sys\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.166098 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-registration-dir\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.166098 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166099    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-host\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.166638 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-multus-conf-dir\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.166638 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166160    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.166638 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-kubelet\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.166638 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166192    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-multus-cni-dir\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.166638 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166178    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 11:51:00.166638 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-systemd-units\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.166638 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-cni-bin\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.166638 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166245    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-multus-conf-dir\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.166638 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166255    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-kubelet\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.166638 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166214    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-registration-dir\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.166638 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.166638 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-systemd-units\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.166638 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-modprobe-d\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.166638 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166309    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-cni-bin\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.166638 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b123f16-31a0-49be-906c-017bdc2602e6-hosts-file\") pod \"node-resolver-4sj9h\" (UID: \"8b123f16-31a0-49be-906c-017bdc2602e6\") " pod="openshift-dns/node-resolver-4sj9h"
Jul 16 11:51:00.167391 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166350    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.167391 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-system-cni-dir\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.167391 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g952t\" (UniqueName: \"kubernetes.io/projected/8b123f16-31a0-49be-906c-017bdc2602e6-kube-api-access-g952t\") pod \"node-resolver-4sj9h\" (UID: \"8b123f16-31a0-49be-906c-017bdc2602e6\") " pod="openshift-dns/node-resolver-4sj9h"
Jul 16 11:51:00.167391 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166398    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b123f16-31a0-49be-906c-017bdc2602e6-hosts-file\") pod \"node-resolver-4sj9h\" (UID: \"8b123f16-31a0-49be-906c-017bdc2602e6\") " pod="openshift-dns/node-resolver-4sj9h"
Jul 16 11:51:00.167391 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166398    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-modprobe-d\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.167391 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-socket-dir\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.167391 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166433    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-system-cni-dir\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.167391 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166546    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-socket-dir\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.167391 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc1f2e2f-da1b-43e6-abd5-87ec82aee7e3-konnectivity-ca\") pod \"konnectivity-agent-mv5xr\" (UID: \"dc1f2e2f-da1b-43e6-abd5-87ec82aee7e3\") " pod="kube-system/konnectivity-agent-mv5xr"
Jul 16 11:51:00.167391 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166581    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.167391 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166595    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-cni-binary-copy\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.167391 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-host-var-lib-cni-bin\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.167391 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166629    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-host-var-lib-cni-bin\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.167391 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166627    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-multus-daemon-config\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.168156 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-host-run-multus-certs\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.168156 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.168156 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb8f8012-4953-426c-8951-cb5dd0c048c1-env-overrides\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.168156 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166771    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-host-run-multus-certs\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.168156 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prlkj\" (UniqueName: \"kubernetes.io/projected/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-kube-api-access-prlkj\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.168156 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166794    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.168156 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-multus-socket-dir-parent\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.168156 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-host-run-k8s-cni-cncf-io\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.168156 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-run-openvswitch\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.168156 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166883    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b123f16-31a0-49be-906c-017bdc2602e6-tmp-dir\") pod \"node-resolver-4sj9h\" (UID: \"8b123f16-31a0-49be-906c-017bdc2602e6\") " pod="openshift-dns/node-resolver-4sj9h"
Jul 16 11:51:00.168156 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166894    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns68k\" (UniqueName: \"kubernetes.io/projected/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-kube-api-access-ns68k\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.168156 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54m8w\" (UniqueName: \"kubernetes.io/projected/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-kube-api-access-54m8w\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.168156 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.168156 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166957    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-host-run-k8s-cni-cncf-io\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.168837 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.166989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-etc-kubernetes\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.168837 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167003    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc1f2e2f-da1b-43e6-abd5-87ec82aee7e3-konnectivity-ca\") pod \"konnectivity-agent-mv5xr\" (UID: \"dc1f2e2f-da1b-43e6-abd5-87ec82aee7e3\") " pod="kube-system/konnectivity-agent-mv5xr"
Jul 16 11:51:00.168837 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-multus-socket-dir-parent\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.168837 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167016    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-var-lib-openvswitch\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.168837 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167053    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-run-openvswitch\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.168837 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167104    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-var-lib-openvswitch\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.168837 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167157    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-etc-kubernetes\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.168837 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167182    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.168837 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-sysctl-d\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.168837 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167220    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb8f8012-4953-426c-8951-cb5dd0c048c1-env-overrides\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.168837 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-sysctl-conf\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.168837 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-var-lib-kubelet\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.168837 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167280    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb8f8012-4953-426c-8951-cb5dd0c048c1-ovnkube-config\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.168837 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-kubelet-config\") pod \"global-pull-secret-syncer-w5wfm\" (UID: \"32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7\") " pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:00.168837 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-var-lib-kubelet\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.169397 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167327    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-device-dir\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.169397 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167348    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-sysctl-d\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.169397 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167364    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-sysctl-conf\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.169397 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-device-dir\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.169397 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-run-netns\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.169397 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167403    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-kubelet-config\") pod \"global-pull-secret-syncer-w5wfm\" (UID: \"32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7\") " pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:00.169397 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs\") pod \"network-metrics-daemon-4p7gl\" (UID: \"18353093-2e17-408e-b541-5dfb339343f0\") " pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:00.169397 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167435    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4lbl\" (UniqueName: \"kubernetes.io/projected/18353093-2e17-408e-b541-5dfb339343f0-kube-api-access-v4lbl\") pod \"network-metrics-daemon-4p7gl\" (UID: \"18353093-2e17-408e-b541-5dfb339343f0\") " pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:00.169397 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-hostroot\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.169397 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167456    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-run-netns\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.169397 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167475    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb8f8012-4953-426c-8951-cb5dd0c048c1-ovnkube-script-lib\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.169397 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/921209ea-e227-46b4-bb01-54ea362d80f0-host-slash\") pod \"iptables-alerter-cl9fk\" (UID: \"921209ea-e227-46b4-bb01-54ea362d80f0\") " pod="openshift-network-operator/iptables-alerter-cl9fk"
Jul 16 11:51:00.169397 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-hostroot\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.169397 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:00.167504    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 11:51:00.169824 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:00.167585    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs podName:18353093-2e17-408e-b541-5dfb339343f0 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:00.667550411 +0000 UTC m=+1.968327510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs") pod "network-metrics-daemon-4p7gl" (UID: "18353093-2e17-408e-b541-5dfb339343f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 11:51:00.169824 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l64k2\" (UniqueName: \"kubernetes.io/projected/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-kube-api-access-l64k2\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.169824 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-cnibin\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.169824 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167643    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-host-var-lib-kubelet\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.169824 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-systemd\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.169824 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167579    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/921209ea-e227-46b4-bb01-54ea362d80f0-host-slash\") pod \"iptables-alerter-cl9fk\" (UID: \"921209ea-e227-46b4-bb01-54ea362d80f0\") " pod="openshift-network-operator/iptables-alerter-cl9fk"
Jul 16 11:51:00.169824 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167699    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-cnibin\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.169824 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-sys-fs\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.169824 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167742    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-systemd\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.169824 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.169824 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-host-var-lib-kubelet\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.169824 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-system-cni-dir\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.169824 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167806    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-slash\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.169824 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167810    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-sys-fs\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.170289 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167833    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-etc-openvswitch\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.170289 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-run-ovn\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.170289 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167864    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb8f8012-4953-426c-8951-cb5dd0c048c1-ovnkube-config\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.170289 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167877    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-system-cni-dir\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.170289 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167891    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-slash\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.170289 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-lib-modules\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.170289 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-etc-openvswitch\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.170289 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-run-ovn\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.170289 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4gmx\" (UniqueName: \"kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx\") pod \"network-check-target-8p6dq\" (UID: \"3768287e-bbac-495d-a015-f6fca7d1ec36\") " pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:00.170289 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.167980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/921209ea-e227-46b4-bb01-54ea362d80f0-iptables-alerter-script\") pod \"iptables-alerter-cl9fk\" (UID: \"921209ea-e227-46b4-bb01-54ea362d80f0\") " pod="openshift-network-operator/iptables-alerter-cl9fk"
Jul 16 11:51:00.170289 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-cni-binary-copy\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.170289 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168032    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret\") pod \"global-pull-secret-syncer-w5wfm\" (UID: \"32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7\") " pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:00.170289 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168042    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb8f8012-4953-426c-8951-cb5dd0c048c1-ovnkube-script-lib\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.170289 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-etc-selinux\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.170289 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-lib-modules\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.170734 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-host-run-netns\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.170734 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-kubernetes\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.170734 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-tmp\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.170734 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:00.168178    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 11:51:00.170734 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168190    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj6m4\" (UniqueName: \"kubernetes.io/projected/921209ea-e227-46b4-bb01-54ea362d80f0-kube-api-access-sj6m4\") pod \"iptables-alerter-cl9fk\" (UID: \"921209ea-e227-46b4-bb01-54ea362d80f0\") " pod="openshift-network-operator/iptables-alerter-cl9fk"
Jul 16 11:51:00.170734 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168210    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-host-run-netns\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.170734 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:00.168234    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret podName:32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:00.668215427 +0000 UTC m=+1.968992521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret") pod "global-pull-secret-syncer-w5wfm" (UID: "32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7") : object "kube-system"/"original-pull-secret" not registered
Jul 16 11:51:00.170734 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-cnibin\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.170734 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-host-var-lib-cni-multus\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.170734 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168312    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-etc-selinux\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.170734 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-run-systemd\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.170734 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168334    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.170734 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb8f8012-4953-426c-8951-cb5dd0c048c1-ovn-node-metrics-cert\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.170734 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168358    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-run-systemd\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.171198 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168383    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57wff\" (UniqueName: \"kubernetes.io/projected/eb8f8012-4953-426c-8951-cb5dd0c048c1-kube-api-access-57wff\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.171198 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168402    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-host-var-lib-cni-multus\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.171198 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-run\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.171198 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-tuned\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.171198 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5d99cb5-959d-4a75-89e7-fa68635e416d-serviceca\") pod \"node-ca-vlbks\" (UID: \"c5d99cb5-959d-4a75-89e7-fa68635e416d\") " pod="openshift-image-registry/node-ca-vlbks"
Jul 16 11:51:00.171198 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168512    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-os-release\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.171198 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-node-log\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.171198 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-cni-netd\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.171198 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-sysconfig\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.171198 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-dbus\") pod \"global-pull-secret-syncer-w5wfm\" (UID: \"32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7\") " pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:00.171198 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168632    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-cni-binary-copy\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.171198 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5d99cb5-959d-4a75-89e7-fa68635e416d-host\") pod \"node-ca-vlbks\" (UID: \"c5d99cb5-959d-4a75-89e7-fa68635e416d\") " pod="openshift-image-registry/node-ca-vlbks"
Jul 16 11:51:00.171198 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkzf5\" (UniqueName: \"kubernetes.io/projected/c5d99cb5-959d-4a75-89e7-fa68635e416d-kube-api-access-rkzf5\") pod \"node-ca-vlbks\" (UID: \"c5d99cb5-959d-4a75-89e7-fa68635e416d\") " pod="openshift-image-registry/node-ca-vlbks"
Jul 16 11:51:00.171198 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168687    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-host-cni-netd\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.171198 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168695    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb8f8012-4953-426c-8951-cb5dd0c048c1-node-log\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.171666 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168388    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-cnibin\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.171666 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168638    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/921209ea-e227-46b4-bb01-54ea362d80f0-iptables-alerter-script\") pod \"iptables-alerter-cl9fk\" (UID: \"921209ea-e227-46b4-bb01-54ea362d80f0\") " pod="openshift-network-operator/iptables-alerter-cl9fk"
Jul 16 11:51:00.171666 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-os-release\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.171666 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-kubernetes\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.171666 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5d99cb5-959d-4a75-89e7-fa68635e416d-host\") pod \"node-ca-vlbks\" (UID: \"c5d99cb5-959d-4a75-89e7-fa68635e416d\") " pod="openshift-image-registry/node-ca-vlbks"
Jul 16 11:51:00.171666 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-sysconfig\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.171666 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168916    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-run\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.171666 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.168970    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-dbus\") pod \"global-pull-secret-syncer-w5wfm\" (UID: \"32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7\") " pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:00.171666 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.169156    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5d99cb5-959d-4a75-89e7-fa68635e416d-serviceca\") pod \"node-ca-vlbks\" (UID: \"c5d99cb5-959d-4a75-89e7-fa68635e416d\") " pod="openshift-image-registry/node-ca-vlbks"
Jul 16 11:51:00.171666 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.170329    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc1f2e2f-da1b-43e6-abd5-87ec82aee7e3-agent-certs\") pod \"konnectivity-agent-mv5xr\" (UID: \"dc1f2e2f-da1b-43e6-abd5-87ec82aee7e3\") " pod="kube-system/konnectivity-agent-mv5xr"
Jul 16 11:51:00.171666 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.170817    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-etc-tuned\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.171666 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.170884    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-tmp\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.171666 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.170958    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb8f8012-4953-426c-8951-cb5dd0c048c1-ovn-node-metrics-cert\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.180695 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:00.180678    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 11:51:00.180695 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:00.180696    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 11:51:00.180836 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:00.180708    2578 projected.go:196] Error preparing data for projected volume kube-api-access-x4gmx for pod openshift-network-diagnostics/network-check-target-8p6dq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 11:51:00.180836 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:00.180766    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx podName:3768287e-bbac-495d-a015-f6fca7d1ec36 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:00.68074714 +0000 UTC m=+1.981524218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x4gmx" (UniqueName: "kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx") pod "network-check-target-8p6dq" (UID: "3768287e-bbac-495d-a015-f6fca7d1ec36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 11:51:00.181262 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.181235    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g952t\" (UniqueName: \"kubernetes.io/projected/8b123f16-31a0-49be-906c-017bdc2602e6-kube-api-access-g952t\") pod \"node-resolver-4sj9h\" (UID: \"8b123f16-31a0-49be-906c-017bdc2602e6\") " pod="openshift-dns/node-resolver-4sj9h"
Jul 16 11:51:00.181719 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.181694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prlkj\" (UniqueName: \"kubernetes.io/projected/28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf-kube-api-access-prlkj\") pod \"tuned-h8jpz\" (UID: \"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.181849 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.181828    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54m8w\" (UniqueName: \"kubernetes.io/projected/7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6-kube-api-access-54m8w\") pod \"multus-z7mwh\" (UID: \"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6\") " pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.183497 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.183474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj6m4\" (UniqueName: \"kubernetes.io/projected/921209ea-e227-46b4-bb01-54ea362d80f0-kube-api-access-sj6m4\") pod \"iptables-alerter-cl9fk\" (UID: \"921209ea-e227-46b4-bb01-54ea362d80f0\") " pod="openshift-network-operator/iptables-alerter-cl9fk"
Jul 16 11:51:00.183580 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.183516    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57wff\" (UniqueName: \"kubernetes.io/projected/eb8f8012-4953-426c-8951-cb5dd0c048c1-kube-api-access-57wff\") pod \"ovnkube-node-kbw5x\" (UID: \"eb8f8012-4953-426c-8951-cb5dd0c048c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.184357 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.184340    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkzf5\" (UniqueName: \"kubernetes.io/projected/c5d99cb5-959d-4a75-89e7-fa68635e416d-kube-api-access-rkzf5\") pod \"node-ca-vlbks\" (UID: \"c5d99cb5-959d-4a75-89e7-fa68635e416d\") " pod="openshift-image-registry/node-ca-vlbks"
Jul 16 11:51:00.184978 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.184957    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4lbl\" (UniqueName: \"kubernetes.io/projected/18353093-2e17-408e-b541-5dfb339343f0-kube-api-access-v4lbl\") pod \"network-metrics-daemon-4p7gl\" (UID: \"18353093-2e17-408e-b541-5dfb339343f0\") " pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:00.185248 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.185227    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns68k\" (UniqueName: \"kubernetes.io/projected/4c9ae01e-f7f2-4296-a0b0-600bd31f36a8-kube-api-access-ns68k\") pod \"multus-additional-cni-plugins-lgflz\" (UID: \"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8\") " pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.186510 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.186487    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l64k2\" (UniqueName: \"kubernetes.io/projected/c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8-kube-api-access-l64k2\") pod \"aws-ebs-csi-driver-node-bjnvz\" (UID: \"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.351122 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.351018    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vlbks"
Jul 16 11:51:00.361618 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.361593    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z7mwh"
Jul 16 11:51:00.369694 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.369666    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cl9fk"
Jul 16 11:51:00.389856 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.389831    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:00.407867 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.407844    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz"
Jul 16 11:51:00.411313 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.411299    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h8jpz"
Jul 16 11:51:00.440396 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.440376    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4sj9h"
Jul 16 11:51:00.453649 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.453622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lgflz"
Jul 16 11:51:00.457050 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:51:00.456994    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b596d2_42ef_47aa_9c6d_8b1a0e2e1ecf.slice/crio-4b8ca7da98f6d2893274aa9a73fbcadc7742aa16e49a1b0c0fbc3b3779ae85e5 WatchSource:0}: Error finding container 4b8ca7da98f6d2893274aa9a73fbcadc7742aa16e49a1b0c0fbc3b3779ae85e5: Status 404 returned error can't find the container with id 4b8ca7da98f6d2893274aa9a73fbcadc7742aa16e49a1b0c0fbc3b3779ae85e5
Jul 16 11:51:00.457881 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.457856    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mv5xr"
Jul 16 11:51:00.458299 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:51:00.458275    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2bb02b0_6901_4c5e_9193_9f7c06c0a8b8.slice/crio-3f53552b2191ccaab26d05beef7f62bab99821997673ba081d6b027ad19f4ede WatchSource:0}: Error finding container 3f53552b2191ccaab26d05beef7f62bab99821997673ba081d6b027ad19f4ede: Status 404 returned error can't find the container with id 3f53552b2191ccaab26d05beef7f62bab99821997673ba081d6b027ad19f4ede
Jul 16 11:51:00.459462 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:51:00.459440    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b123f16_31a0_49be_906c_017bdc2602e6.slice/crio-91b213a570ab684a15a439adecac2978be21daf42fbf9e55ba453cb7a80af1f5 WatchSource:0}: Error finding container 91b213a570ab684a15a439adecac2978be21daf42fbf9e55ba453cb7a80af1f5: Status 404 returned error can't find the container with id 91b213a570ab684a15a439adecac2978be21daf42fbf9e55ba453cb7a80af1f5
Jul 16 11:51:00.460257 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:51:00.460234    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84d25049db764d4ac62163954d83d46.slice/crio-54dc347d3fa6eac7979f919c819749672828d6133a18faba6b9e9aaee33c19e6 WatchSource:0}: Error finding container 54dc347d3fa6eac7979f919c819749672828d6133a18faba6b9e9aaee33c19e6: Status 404 returned error can't find the container with id 54dc347d3fa6eac7979f919c819749672828d6133a18faba6b9e9aaee33c19e6
Jul 16 11:51:00.466151 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.466120    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 11:51:00.469193 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:51:00.469170    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9ae01e_f7f2_4296_a0b0_600bd31f36a8.slice/crio-ee9e9a9fcc7f6c74b5438a12bc97e35a099b10d642a702f09ab5c4ce8a5fdd5b WatchSource:0}: Error finding container ee9e9a9fcc7f6c74b5438a12bc97e35a099b10d642a702f09ab5c4ce8a5fdd5b: Status 404 returned error can't find the container with id ee9e9a9fcc7f6c74b5438a12bc97e35a099b10d642a702f09ab5c4ce8a5fdd5b
Jul 16 11:51:00.470208 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:51:00.470187    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1f2e2f_da1b_43e6_abd5_87ec82aee7e3.slice/crio-00b1c3ee8a13a16ac3ed6a761703195240d86c69dca1985e3bef53583444a5a6 WatchSource:0}: Error finding container 00b1c3ee8a13a16ac3ed6a761703195240d86c69dca1985e3bef53583444a5a6: Status 404 returned error can't find the container with id 00b1c3ee8a13a16ac3ed6a761703195240d86c69dca1985e3bef53583444a5a6
Jul 16 11:51:00.504361 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:51:00.504337    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c1f669f09517dfcf8e52b12af2bb23.slice/crio-c211e60bc7b898684ede25e8a7bf083c23da87dca6db81a0ffea5aa6edb85dc7 WatchSource:0}: Error finding container c211e60bc7b898684ede25e8a7bf083c23da87dca6db81a0ffea5aa6edb85dc7: Status 404 returned error can't find the container with id c211e60bc7b898684ede25e8a7bf083c23da87dca6db81a0ffea5aa6edb85dc7
Jul 16 11:51:00.671837 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.671768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs\") pod \"network-metrics-daemon-4p7gl\" (UID: \"18353093-2e17-408e-b541-5dfb339343f0\") " pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:00.671837 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.671826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret\") pod \"global-pull-secret-syncer-w5wfm\" (UID: \"32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7\") " pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:00.672004 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:00.671893    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 11:51:00.672004 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:00.671929    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 11:51:00.672004 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:00.671947    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs podName:18353093-2e17-408e-b541-5dfb339343f0 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:01.671931434 +0000 UTC m=+2.972708511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs") pod "network-metrics-daemon-4p7gl" (UID: "18353093-2e17-408e-b541-5dfb339343f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 11:51:00.672004 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:00.671982    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret podName:32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:01.671964408 +0000 UTC m=+2.972741504 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret") pod "global-pull-secret-syncer-w5wfm" (UID: "32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7") : object "kube-system"/"original-pull-secret" not registered
Jul 16 11:51:00.768468 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:51:00.768436    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfeaf25_6b49_467f_ae5d_5e0ea9c503e6.slice/crio-3d99b1c920100a6758c03663d3e31a99a0abfd1b1756dcb9cc022e1646cc9a81 WatchSource:0}: Error finding container 3d99b1c920100a6758c03663d3e31a99a0abfd1b1756dcb9cc022e1646cc9a81: Status 404 returned error can't find the container with id 3d99b1c920100a6758c03663d3e31a99a0abfd1b1756dcb9cc022e1646cc9a81
Jul 16 11:51:00.772849 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.772827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4gmx\" (UniqueName: \"kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx\") pod \"network-check-target-8p6dq\" (UID: \"3768287e-bbac-495d-a015-f6fca7d1ec36\") " pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:00.772957 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:00.772945    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 11:51:00.773004 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:00.772960    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 11:51:00.773004 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:00.772970    2578 projected.go:196] Error preparing data for projected volume kube-api-access-x4gmx for pod openshift-network-diagnostics/network-check-target-8p6dq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 11:51:00.773094 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:00.773011    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx podName:3768287e-bbac-495d-a015-f6fca7d1ec36 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:01.772998309 +0000 UTC m=+3.073775386 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x4gmx" (UniqueName: "kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx") pod "network-check-target-8p6dq" (UID: "3768287e-bbac-495d-a015-f6fca7d1ec36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 11:51:00.808872 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:51:00.808846    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d99cb5_959d_4a75_89e7_fa68635e416d.slice/crio-32258ccc3608c93babc6a539873cfa624c332c972608e9d2b47459db23bc4cce WatchSource:0}: Error finding container 32258ccc3608c93babc6a539873cfa624c332c972608e9d2b47459db23bc4cce: Status 404 returned error can't find the container with id 32258ccc3608c93babc6a539873cfa624c332c972608e9d2b47459db23bc4cce
Jul 16 11:51:00.832192 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:00.832162    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 11:51:00.852941 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:51:00.852919    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod921209ea_e227_46b4_bb01_54ea362d80f0.slice/crio-0ead0926531b39d31940579f00ffd38b35dc56a2b300e9552f4a7530b3e9b389 WatchSource:0}: Error finding container 0ead0926531b39d31940579f00ffd38b35dc56a2b300e9552f4a7530b3e9b389: Status 404 returned error can't find the container with id 0ead0926531b39d31940579f00ffd38b35dc56a2b300e9552f4a7530b3e9b389
Jul 16 11:51:01.119849 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:01.119795    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 11:46:00 +0000 UTC" deadline="2027-12-16 17:32:45.261582643 +0000 UTC"
Jul 16 11:51:01.119849 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:01.119844    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12437h41m44.141743876s"
Jul 16 11:51:01.307134 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:01.302000    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4sj9h" event={"ID":"8b123f16-31a0-49be-906c-017bdc2602e6","Type":"ContainerStarted","Data":"91b213a570ab684a15a439adecac2978be21daf42fbf9e55ba453cb7a80af1f5"}
Jul 16 11:51:01.307134 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:01.305852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz" event={"ID":"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8","Type":"ContainerStarted","Data":"3f53552b2191ccaab26d05beef7f62bab99821997673ba081d6b027ad19f4ede"}
Jul 16 11:51:01.308657 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:01.308625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h8jpz" event={"ID":"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf","Type":"ContainerStarted","Data":"4b8ca7da98f6d2893274aa9a73fbcadc7742aa16e49a1b0c0fbc3b3779ae85e5"}
Jul 16 11:51:01.314571 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:01.314543    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mv5xr" event={"ID":"dc1f2e2f-da1b-43e6-abd5-87ec82aee7e3","Type":"ContainerStarted","Data":"00b1c3ee8a13a16ac3ed6a761703195240d86c69dca1985e3bef53583444a5a6"}
Jul 16 11:51:01.323775 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:01.323724    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal" event={"ID":"f84d25049db764d4ac62163954d83d46","Type":"ContainerStarted","Data":"54dc347d3fa6eac7979f919c819749672828d6133a18faba6b9e9aaee33c19e6"}
Jul 16 11:51:01.334076 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:01.334030    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cl9fk" event={"ID":"921209ea-e227-46b4-bb01-54ea362d80f0","Type":"ContainerStarted","Data":"0ead0926531b39d31940579f00ffd38b35dc56a2b300e9552f4a7530b3e9b389"}
Jul 16 11:51:01.346374 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:01.346346    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vlbks" event={"ID":"c5d99cb5-959d-4a75-89e7-fa68635e416d","Type":"ContainerStarted","Data":"32258ccc3608c93babc6a539873cfa624c332c972608e9d2b47459db23bc4cce"}
Jul 16 11:51:01.352771 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:01.352746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z7mwh" event={"ID":"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6","Type":"ContainerStarted","Data":"3d99b1c920100a6758c03663d3e31a99a0abfd1b1756dcb9cc022e1646cc9a81"}
Jul 16 11:51:01.363547 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:01.363495    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal" event={"ID":"84c1f669f09517dfcf8e52b12af2bb23","Type":"ContainerStarted","Data":"c211e60bc7b898684ede25e8a7bf083c23da87dca6db81a0ffea5aa6edb85dc7"}
Jul 16 11:51:01.367476 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:01.367441    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgflz" event={"ID":"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8","Type":"ContainerStarted","Data":"ee9e9a9fcc7f6c74b5438a12bc97e35a099b10d642a702f09ab5c4ce8a5fdd5b"}
Jul 16 11:51:01.530753 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:01.530498    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 11:51:01.639081 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:01.638827    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 11:51:01.681013 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:01.680975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret\") pod \"global-pull-secret-syncer-w5wfm\" (UID: \"32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7\") " pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:01.681273 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:01.681074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs\") pod \"network-metrics-daemon-4p7gl\" (UID: \"18353093-2e17-408e-b541-5dfb339343f0\") " pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:01.681273 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:01.681215    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 11:51:01.681406 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:01.681279    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs podName:18353093-2e17-408e-b541-5dfb339343f0 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:03.681258324 +0000 UTC m=+4.982035416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs") pod "network-metrics-daemon-4p7gl" (UID: "18353093-2e17-408e-b541-5dfb339343f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 11:51:01.681704 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:01.681685    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 11:51:01.681774 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:01.681742    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret podName:32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:03.681725739 +0000 UTC m=+4.982502834 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret") pod "global-pull-secret-syncer-w5wfm" (UID: "32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7") : object "kube-system"/"original-pull-secret" not registered
Jul 16 11:51:01.782363 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:01.782257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4gmx\" (UniqueName: \"kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx\") pod \"network-check-target-8p6dq\" (UID: \"3768287e-bbac-495d-a015-f6fca7d1ec36\") " pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:01.782524 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:01.782449    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 11:51:01.782524 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:01.782468    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 11:51:01.782524 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:01.782480    2578 projected.go:196] Error preparing data for projected volume kube-api-access-x4gmx for pod openshift-network-diagnostics/network-check-target-8p6dq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 11:51:01.782684 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:01.782534    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx podName:3768287e-bbac-495d-a015-f6fca7d1ec36 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:03.782513958 +0000 UTC m=+5.083291050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x4gmx" (UniqueName: "kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx") pod "network-check-target-8p6dq" (UID: "3768287e-bbac-495d-a015-f6fca7d1ec36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 11:51:02.120520 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:02.120413    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 11:46:00 +0000 UTC" deadline="2027-12-31 22:02:02.577987904 +0000 UTC"
Jul 16 11:51:02.120520 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:02.120488    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12802h11m0.457504645s"
Jul 16 11:51:02.294431 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:02.294397    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:02.294622 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:02.294399    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:02.294622 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:02.294536    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:02.294622 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:02.294530    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5wfm" podUID="32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7"
Jul 16 11:51:02.294797 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:02.294634    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p6dq" podUID="3768287e-bbac-495d-a015-f6fca7d1ec36"
Jul 16 11:51:02.294797 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:02.294720    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p7gl" podUID="18353093-2e17-408e-b541-5dfb339343f0"
Jul 16 11:51:02.383794 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:02.383690    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x" event={"ID":"eb8f8012-4953-426c-8951-cb5dd0c048c1","Type":"ContainerStarted","Data":"f73b681e7c35fcafe1691242274494e41517140470a2b2faf32a88ccc55f62b9"}
Jul 16 11:51:03.699880 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:03.699842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs\") pod \"network-metrics-daemon-4p7gl\" (UID: \"18353093-2e17-408e-b541-5dfb339343f0\") " pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:03.700424 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:03.699921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret\") pod \"global-pull-secret-syncer-w5wfm\" (UID: \"32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7\") " pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:03.700424 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:03.700098    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 11:51:03.700424 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:03.700164    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret podName:32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:07.70014276 +0000 UTC m=+9.000919857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret") pod "global-pull-secret-syncer-w5wfm" (UID: "32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7") : object "kube-system"/"original-pull-secret" not registered
Jul 16 11:51:03.700632 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:03.700594    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 11:51:03.700692 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:03.700647    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs podName:18353093-2e17-408e-b541-5dfb339343f0 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:07.700629225 +0000 UTC m=+9.001406320 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs") pod "network-metrics-daemon-4p7gl" (UID: "18353093-2e17-408e-b541-5dfb339343f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 11:51:03.800900 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:03.800860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4gmx\" (UniqueName: \"kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx\") pod \"network-check-target-8p6dq\" (UID: \"3768287e-bbac-495d-a015-f6fca7d1ec36\") " pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:03.801185 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:03.801089    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 11:51:03.801185 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:03.801115    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 11:51:03.801185 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:03.801127    2578 projected.go:196] Error preparing data for projected volume kube-api-access-x4gmx for pod openshift-network-diagnostics/network-check-target-8p6dq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 11:51:03.801395 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:03.801191    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx podName:3768287e-bbac-495d-a015-f6fca7d1ec36 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:07.801169708 +0000 UTC m=+9.101946790 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x4gmx" (UniqueName: "kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx") pod "network-check-target-8p6dq" (UID: "3768287e-bbac-495d-a015-f6fca7d1ec36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 11:51:04.293350 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:04.293316    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:04.293526 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:04.293461    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5wfm" podUID="32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7"
Jul 16 11:51:04.293876 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:04.293845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:04.293970 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:04.293945    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p6dq" podUID="3768287e-bbac-495d-a015-f6fca7d1ec36"
Jul 16 11:51:04.294043 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:04.294002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:04.294135 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:04.294091    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p7gl" podUID="18353093-2e17-408e-b541-5dfb339343f0"
Jul 16 11:51:06.293542 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:06.293479    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:06.294017 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:06.293580    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:06.294017 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:06.293616    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:06.294017 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:06.293724    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p6dq" podUID="3768287e-bbac-495d-a015-f6fca7d1ec36"
Jul 16 11:51:06.294353 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:06.294199    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p7gl" podUID="18353093-2e17-408e-b541-5dfb339343f0"
Jul 16 11:51:06.294353 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:06.294308    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5wfm" podUID="32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7"
Jul 16 11:51:07.738633 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:07.738454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs\") pod \"network-metrics-daemon-4p7gl\" (UID: \"18353093-2e17-408e-b541-5dfb339343f0\") " pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:07.738633 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:07.738543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret\") pod \"global-pull-secret-syncer-w5wfm\" (UID: \"32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7\") " pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:07.738633 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:07.738616    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 11:51:07.739297 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:07.738649    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 11:51:07.739297 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:07.738699    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs podName:18353093-2e17-408e-b541-5dfb339343f0 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:15.738675362 +0000 UTC m=+17.039452455 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs") pod "network-metrics-daemon-4p7gl" (UID: "18353093-2e17-408e-b541-5dfb339343f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 11:51:07.739297 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:07.738726    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret podName:32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:15.738712024 +0000 UTC m=+17.039489111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret") pod "global-pull-secret-syncer-w5wfm" (UID: "32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7") : object "kube-system"/"original-pull-secret" not registered
Jul 16 11:51:07.839286 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:07.839244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4gmx\" (UniqueName: \"kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx\") pod \"network-check-target-8p6dq\" (UID: \"3768287e-bbac-495d-a015-f6fca7d1ec36\") " pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:07.839452 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:07.839433    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 11:51:07.839512 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:07.839457    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 11:51:07.839512 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:07.839471    2578 projected.go:196] Error preparing data for projected volume kube-api-access-x4gmx for pod openshift-network-diagnostics/network-check-target-8p6dq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 11:51:07.839628 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:07.839543    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx podName:3768287e-bbac-495d-a015-f6fca7d1ec36 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:15.839523644 +0000 UTC m=+17.140300741 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x4gmx" (UniqueName: "kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx") pod "network-check-target-8p6dq" (UID: "3768287e-bbac-495d-a015-f6fca7d1ec36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 11:51:08.294103 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:08.294021    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:08.294103 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:08.294044    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:08.294103 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:08.294027    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:08.294391 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:08.294159    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5wfm" podUID="32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7"
Jul 16 11:51:08.294721 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:08.294607    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p6dq" podUID="3768287e-bbac-495d-a015-f6fca7d1ec36"
Jul 16 11:51:08.294823 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:08.294722    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p7gl" podUID="18353093-2e17-408e-b541-5dfb339343f0"
Jul 16 11:51:10.293997 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:10.293961    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:10.294457 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:10.294150    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5wfm" podUID="32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7"
Jul 16 11:51:10.294457 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:10.294221    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:10.294457 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:10.294334    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p6dq" podUID="3768287e-bbac-495d-a015-f6fca7d1ec36"
Jul 16 11:51:10.294457 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:10.294383    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:10.294673 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:10.294464    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p7gl" podUID="18353093-2e17-408e-b541-5dfb339343f0"
Jul 16 11:51:12.293452 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:12.293422    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:12.293452 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:12.293451    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:12.293924 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:12.293421    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:12.293924 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:12.293532    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p7gl" podUID="18353093-2e17-408e-b541-5dfb339343f0"
Jul 16 11:51:12.293924 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:12.293616    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p6dq" podUID="3768287e-bbac-495d-a015-f6fca7d1ec36"
Jul 16 11:51:12.293924 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:12.293723    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5wfm" podUID="32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7"
Jul 16 11:51:14.293866 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:14.293824    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:14.294331 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:14.293950    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:14.294331 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:14.293961    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p7gl" podUID="18353093-2e17-408e-b541-5dfb339343f0"
Jul 16 11:51:14.294331 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:14.294054    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5wfm" podUID="32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7"
Jul 16 11:51:14.294331 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:14.294062    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:14.294331 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:14.294143    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p6dq" podUID="3768287e-bbac-495d-a015-f6fca7d1ec36"
Jul 16 11:51:15.805678 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:15.805632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs\") pod \"network-metrics-daemon-4p7gl\" (UID: \"18353093-2e17-408e-b541-5dfb339343f0\") " pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:15.806117 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:15.805699    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret\") pod \"global-pull-secret-syncer-w5wfm\" (UID: \"32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7\") " pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:15.806117 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:15.805817    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 11:51:15.806117 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:15.805893    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs podName:18353093-2e17-408e-b541-5dfb339343f0 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:31.805871416 +0000 UTC m=+33.106648503 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs") pod "network-metrics-daemon-4p7gl" (UID: "18353093-2e17-408e-b541-5dfb339343f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 11:51:15.806117 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:15.805827    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 11:51:15.806117 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:15.806010    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret podName:32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:31.805988435 +0000 UTC m=+33.106765525 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret") pod "global-pull-secret-syncer-w5wfm" (UID: "32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7") : object "kube-system"/"original-pull-secret" not registered
Jul 16 11:51:15.906601 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:15.906564    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4gmx\" (UniqueName: \"kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx\") pod \"network-check-target-8p6dq\" (UID: \"3768287e-bbac-495d-a015-f6fca7d1ec36\") " pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:15.906770 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:15.906738    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 11:51:15.906770 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:15.906765    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 11:51:15.906864 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:15.906780    2578 projected.go:196] Error preparing data for projected volume kube-api-access-x4gmx for pod openshift-network-diagnostics/network-check-target-8p6dq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 11:51:15.906864 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:15.906836    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx podName:3768287e-bbac-495d-a015-f6fca7d1ec36 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:31.90682125 +0000 UTC m=+33.207598349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x4gmx" (UniqueName: "kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx") pod "network-check-target-8p6dq" (UID: "3768287e-bbac-495d-a015-f6fca7d1ec36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 11:51:16.293588 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:16.293557    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:16.293766 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:16.293590    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:16.293766 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:16.293658    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:16.293766 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:16.293669    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5wfm" podUID="32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7"
Jul 16 11:51:16.293766 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:16.293725    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p6dq" podUID="3768287e-bbac-495d-a015-f6fca7d1ec36"
Jul 16 11:51:16.293969 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:16.293805    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p7gl" podUID="18353093-2e17-408e-b541-5dfb339343f0"
Jul 16 11:51:18.294118 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:18.293903    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:18.294563 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:18.293904    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:18.294563 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:18.294227    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p7gl" podUID="18353093-2e17-408e-b541-5dfb339343f0"
Jul 16 11:51:18.294563 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:18.294316    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5wfm" podUID="32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7"
Jul 16 11:51:18.294563 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:18.293899    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:18.294563 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:18.294411    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p6dq" podUID="3768287e-bbac-495d-a015-f6fca7d1ec36"
Jul 16 11:51:20.293392 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:20.293356    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:20.293822 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:20.293403    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:20.293822 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:20.293453    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:20.293822 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:20.293463    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p6dq" podUID="3768287e-bbac-495d-a015-f6fca7d1ec36"
Jul 16 11:51:20.293822 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:20.293609    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p7gl" podUID="18353093-2e17-408e-b541-5dfb339343f0"
Jul 16 11:51:20.293822 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:20.293678    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5wfm" podUID="32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7"
Jul 16 11:51:22.294269 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.294136    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:22.295055 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:22.294338    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p6dq" podUID="3768287e-bbac-495d-a015-f6fca7d1ec36"
Jul 16 11:51:22.295055 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.294136    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:22.295055 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:22.294399    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5wfm" podUID="32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7"
Jul 16 11:51:22.295055 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.294170    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:22.295055 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:22.294449    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p7gl" podUID="18353093-2e17-408e-b541-5dfb339343f0"
Jul 16 11:51:22.427761 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.427732    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal" event={"ID":"84c1f669f09517dfcf8e52b12af2bb23","Type":"ContainerStarted","Data":"1b4f8bb92526841dcfc3bd5c268a3677bd7239beee17bef342c5e8d035c7b793"}
Jul 16 11:51:22.429150 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.429127    2578 generic.go:356] "Generic (PLEG): container finished" podID="4c9ae01e-f7f2-4296-a0b0-600bd31f36a8" containerID="2daf64b0fbfa72a5b6a9ebae34845c8d54ac70e5363cf83c0998d735707849dc" exitCode=0
Jul 16 11:51:22.429234 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.429195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgflz" event={"ID":"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8","Type":"ContainerDied","Data":"2daf64b0fbfa72a5b6a9ebae34845c8d54ac70e5363cf83c0998d735707849dc"}
Jul 16 11:51:22.430647 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.430619    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4sj9h" event={"ID":"8b123f16-31a0-49be-906c-017bdc2602e6","Type":"ContainerStarted","Data":"958c2fa5460787d19ef2017b73c5a1b0e0b922f5f447f7cd6da56045eb2fc422"}
Jul 16 11:51:22.432147 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.432127    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz" event={"ID":"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8","Type":"ContainerStarted","Data":"53e8167dbeee1b5d2eaeccdca82e9b40e2a1e2b1e33e30e42e7caefe010ff825"}
Jul 16 11:51:22.433676 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.433654    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h8jpz" event={"ID":"28b596d2-42ef-47aa-9c6d-8b1a0e2e1ecf","Type":"ContainerStarted","Data":"20627ea1decc92b617592d4e3376494e4b92c7fb89e026e2670dbd4bb1fd141b"}
Jul 16 11:51:22.435059 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.435014    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mv5xr" event={"ID":"dc1f2e2f-da1b-43e6-abd5-87ec82aee7e3","Type":"ContainerStarted","Data":"6a967f86af35cee70982b424cf1c6bf6218be79562a6b3227fe7d5d32a1568f1"}
Jul 16 11:51:22.436503 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.436482    2578 generic.go:356] "Generic (PLEG): container finished" podID="f84d25049db764d4ac62163954d83d46" containerID="0413a0056179a98f330ea9504461ea2b7b34a98a1961a3bc374c307d2633231a" exitCode=0
Jul 16 11:51:22.436590 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.436554    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal" event={"ID":"f84d25049db764d4ac62163954d83d46","Type":"ContainerDied","Data":"0413a0056179a98f330ea9504461ea2b7b34a98a1961a3bc374c307d2633231a"}
Jul 16 11:51:22.438912 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.438890    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x" event={"ID":"eb8f8012-4953-426c-8951-cb5dd0c048c1","Type":"ContainerStarted","Data":"e028644574609b48750a1e76ef32507813a24a70929e7ab2c1f993c04eaeebcf"}
Jul 16 11:51:22.439024 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.438974    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x" event={"ID":"eb8f8012-4953-426c-8951-cb5dd0c048c1","Type":"ContainerStarted","Data":"c9799151970fd251ffcae5334c1049e74722b6eb7c2627d350c1e664dda692e9"}
Jul 16 11:51:22.439092 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.439024    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x" event={"ID":"eb8f8012-4953-426c-8951-cb5dd0c048c1","Type":"ContainerStarted","Data":"90c4a735d30bb9263c87f9effc95b820b6411fbf76ba54fff851b6e0dc6d44a1"}
Jul 16 11:51:22.439169 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.439093    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x" event={"ID":"eb8f8012-4953-426c-8951-cb5dd0c048c1","Type":"ContainerStarted","Data":"1c31132540ee1f45c165d39c5d4e91b0c2813cdba26d8901195758ce033d9414"}
Jul 16 11:51:22.440371 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.440353    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vlbks" event={"ID":"c5d99cb5-959d-4a75-89e7-fa68635e416d","Type":"ContainerStarted","Data":"423f57d565dcea1a00b73060fba1d01fa4121a62fb6965ea4ce687b220ca1449"}
Jul 16 11:51:22.441194 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.441155    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal" podStartSLOduration=22.441141918 podStartE2EDuration="22.441141918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:51:22.44094713 +0000 UTC m=+23.741724229" watchObservedRunningTime="2026-07-16 11:51:22.441141918 +0000 UTC m=+23.741919016"
Jul 16 11:51:22.441686 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.441660    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z7mwh" event={"ID":"7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6","Type":"ContainerStarted","Data":"71b55f49f591311e939d109b90c046f2052cf6d94d833e67a510cbbfa75eb2b7"}
Jul 16 11:51:22.454027 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.453995    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mv5xr" podStartSLOduration=1.088792358 podStartE2EDuration="22.453984003s" totalImagesPullingTime="21.365191645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:51:22.453598245 +0000 UTC m=+23.754375343" watchObservedRunningTime="2026-07-16 11:51:22.453984003 +0000 UTC m=+23.754761101"
Jul 16 11:51:22.482753 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.482695    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h8jpz" podStartSLOduration=1.08692965 podStartE2EDuration="22.482683215s" totalImagesPullingTime="21.395753565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:51:22.469015108 +0000 UTC m=+23.769792207" watchObservedRunningTime="2026-07-16 11:51:22.482683215 +0000 UTC m=+23.783460338"
Jul 16 11:51:22.482828 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.482779    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4sj9h" podStartSLOduration=1.110865665 podStartE2EDuration="22.482773082s" totalImagesPullingTime="21.371907417s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:51:22.482287342 +0000 UTC m=+23.783064440" watchObservedRunningTime="2026-07-16 11:51:22.482773082 +0000 UTC m=+23.783550181"
Jul 16 11:51:22.536804 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:22.536771    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vlbks" podStartSLOduration=1.5102354550000001 podStartE2EDuration="22.536758312s" totalImagesPullingTime="21.026522857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:51:22.515841508 +0000 UTC m=+23.816618605" watchObservedRunningTime="2026-07-16 11:51:22.536758312 +0000 UTC m=+23.837535471"
Jul 16 11:51:23.445283 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:23.445019    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cl9fk" event={"ID":"921209ea-e227-46b4-bb01-54ea362d80f0","Type":"ContainerStarted","Data":"3aef08e4133b8029077b310d9b0130d56b739edcb9dcfe7e651075ff7e09a1fc"}
Jul 16 11:51:23.448629 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:23.448600    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x" event={"ID":"eb8f8012-4953-426c-8951-cb5dd0c048c1","Type":"ContainerStarted","Data":"de5e71e2532d94d01c168607765696e98297c759e88ef302f1582bc9a1caccaf"}
Jul 16 11:51:23.448789 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:23.448678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x" event={"ID":"eb8f8012-4953-426c-8951-cb5dd0c048c1","Type":"ContainerStarted","Data":"6db2f5e675a7cbba964e1f03ce40d97682680c4b4647bc6cca70cab16e1f3617"}
Jul 16 11:51:23.459262 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:23.459214    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-z7mwh" podStartSLOduration=2.337797985 podStartE2EDuration="23.459199733s" totalImagesPullingTime="21.121401748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:51:22.536878252 +0000 UTC m=+23.837655342" watchObservedRunningTime="2026-07-16 11:51:23.459199733 +0000 UTC m=+24.759976833"
Jul 16 11:51:23.459383 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:23.459361    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cl9fk" podStartSLOduration=2.476905166 podStartE2EDuration="23.459355855s" totalImagesPullingTime="20.982450689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:51:23.458862037 +0000 UTC m=+24.759639139" watchObservedRunningTime="2026-07-16 11:51:23.459355855 +0000 UTC m=+24.760132953"
Jul 16 11:51:23.612208 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:23.612177    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mv5xr"
Jul 16 11:51:23.612751 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:23.612732    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mv5xr"
Jul 16 11:51:24.293554 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:24.293509    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:24.293554 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:24.293549    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:24.293872 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:24.293506    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:24.293872 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:24.293636    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p7gl" podUID="18353093-2e17-408e-b541-5dfb339343f0"
Jul 16 11:51:24.293872 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:24.293695    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5wfm" podUID="32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7"
Jul 16 11:51:24.293872 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:24.293808    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p6dq" podUID="3768287e-bbac-495d-a015-f6fca7d1ec36"
Jul 16 11:51:24.453431 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:24.453389    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal" event={"ID":"f84d25049db764d4ac62163954d83d46","Type":"ContainerStarted","Data":"e8df28f47de39671c77bcdecee0f0c01b02469af4912b0e1db5575c52774e47d"}
Jul 16 11:51:24.469027 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:24.468984    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal" podStartSLOduration=24.468964299 podStartE2EDuration="24.468964299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:51:24.468827188 +0000 UTC m=+25.769604285" watchObservedRunningTime="2026-07-16 11:51:24.468964299 +0000 UTC m=+25.769741398"
Jul 16 11:51:24.774803 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:24.774717    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 11:51:25.180479 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:25.180386    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T11:51:24.774736801Z","UUID":"c6d4bdce-a515-44d5-ba9b-1ea7a298875e","Handler":null,"Name":"","Endpoint":""}
Jul 16 11:51:25.182295 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:25.182275    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 11:51:25.182429 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:25.182312    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 11:51:25.458516 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:25.458416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz" event={"ID":"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8","Type":"ContainerStarted","Data":"5ce4b97e35a649aefb766502b3f97f0e66e967968e66c05f56a0975013ac4d01"}
Jul 16 11:51:25.462336 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:25.462295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x" event={"ID":"eb8f8012-4953-426c-8951-cb5dd0c048c1","Type":"ContainerStarted","Data":"89bfe7baf39357491ac60a9f42f013ac7eda3658b2d8f5ca9b9559eb13a8ff8b"}
Jul 16 11:51:25.462484 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:25.462344    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 16 11:51:26.294007 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:26.293972    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:26.294197 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:26.293981    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:26.294197 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:26.294099    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:26.294197 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:26.294096    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5wfm" podUID="32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7"
Jul 16 11:51:26.294197 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:26.294165    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p6dq" podUID="3768287e-bbac-495d-a015-f6fca7d1ec36"
Jul 16 11:51:26.294362 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:26.294268    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p7gl" podUID="18353093-2e17-408e-b541-5dfb339343f0"
Jul 16 11:51:27.468289 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:27.468127    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz" event={"ID":"c2bb02b0-6901-4c5e-9193-9f7c06c0a8b8","Type":"ContainerStarted","Data":"49f7134427296f2b2f0955e469fc44277334fedd77fb7f864742967b872c04dd"}
Jul 16 11:51:27.471588 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:27.471555    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x" event={"ID":"eb8f8012-4953-426c-8951-cb5dd0c048c1","Type":"ContainerStarted","Data":"844ba0beed22686babaded62491a9be6477a804db1d5476cb5fcbcef9621c615"}
Jul 16 11:51:27.471791 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:27.471770    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:27.471866 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:27.471826    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:27.473427 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:27.473403    2578 generic.go:356] "Generic (PLEG): container finished" podID="4c9ae01e-f7f2-4296-a0b0-600bd31f36a8" containerID="8c4dfc25b23687c5eb195d78bc233c2f6c463f6609f9d2fc78105c03a12746fe" exitCode=0
Jul 16 11:51:27.473518 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:27.473437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgflz" event={"ID":"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8","Type":"ContainerDied","Data":"8c4dfc25b23687c5eb195d78bc233c2f6c463f6609f9d2fc78105c03a12746fe"}
Jul 16 11:51:27.485396 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:27.485356    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bjnvz" podStartSLOduration=1.27868713 podStartE2EDuration="27.485343503s" totalImagesPullingTime="26.206656373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:51:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:51:27.484857716 +0000 UTC m=+28.785634808" watchObservedRunningTime="2026-07-16 11:51:27.485343503 +0000 UTC m=+28.786120600"
Jul 16 11:51:27.486274 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:27.486257    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:27.528237 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:27.528200    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x" podStartSLOduration=7.521067953 podStartE2EDuration="27.52818566s" totalImagesPullingTime="20.007117707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:51:27.527773839 +0000 UTC m=+28.828550940" watchObservedRunningTime="2026-07-16 11:51:27.52818566 +0000 UTC m=+28.828962758"
Jul 16 11:51:27.570294 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:27.570272    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mv5xr"
Jul 16 11:51:27.570418 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:27.570406    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 16 11:51:27.570932 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:27.570910    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mv5xr"
Jul 16 11:51:28.293460 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:28.293429    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:28.293777 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:28.293430    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:28.293777 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:28.293537    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p7gl" podUID="18353093-2e17-408e-b541-5dfb339343f0"
Jul 16 11:51:28.293777 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:28.293430    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:28.293777 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:28.293599    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p6dq" podUID="3768287e-bbac-495d-a015-f6fca7d1ec36"
Jul 16 11:51:28.293777 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:28.293711    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5wfm" podUID="32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7"
Jul 16 11:51:28.475759 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:28.475620    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:28.489450 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:28.489422    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:51:28.927710 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:28.927639    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w5wfm"]
Jul 16 11:51:28.927914 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:28.927733    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:28.927914 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:28.927813    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5wfm" podUID="32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7"
Jul 16 11:51:28.931077 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:28.931052    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4p7gl"]
Jul 16 11:51:28.931221 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:28.931150    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:28.931280 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:28.931233    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p7gl" podUID="18353093-2e17-408e-b541-5dfb339343f0"
Jul 16 11:51:28.931676 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:28.931656    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8p6dq"]
Jul 16 11:51:28.931763 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:28.931724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:28.931811 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:28.931798    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p6dq" podUID="3768287e-bbac-495d-a015-f6fca7d1ec36"
Jul 16 11:51:29.479320 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:29.479283    2578 generic.go:356] "Generic (PLEG): container finished" podID="4c9ae01e-f7f2-4296-a0b0-600bd31f36a8" containerID="eb7819ccce3db5f296b749b8f1835a66e778582f8bd425ad67d48aba51c18892" exitCode=0
Jul 16 11:51:29.479678 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:29.479371    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgflz" event={"ID":"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8","Type":"ContainerDied","Data":"eb7819ccce3db5f296b749b8f1835a66e778582f8bd425ad67d48aba51c18892"}
Jul 16 11:51:30.293861 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:30.293830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:30.294090 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:30.293837    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:30.294090 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:30.293958    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5wfm" podUID="32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7"
Jul 16 11:51:30.294090 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:30.294003    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p6dq" podUID="3768287e-bbac-495d-a015-f6fca7d1ec36"
Jul 16 11:51:31.293966 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:31.293935    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:31.294406 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:31.294096    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p7gl" podUID="18353093-2e17-408e-b541-5dfb339343f0"
Jul 16 11:51:31.486253 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:31.486218    2578 generic.go:356] "Generic (PLEG): container finished" podID="4c9ae01e-f7f2-4296-a0b0-600bd31f36a8" containerID="3b3632596937eec49126a315f3496812db9d3d8da787d8441f06b9d2d658a1d3" exitCode=0
Jul 16 11:51:31.486420 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:31.486284    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgflz" event={"ID":"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8","Type":"ContainerDied","Data":"3b3632596937eec49126a315f3496812db9d3d8da787d8441f06b9d2d658a1d3"}
Jul 16 11:51:31.821151 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:31.821070    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs\") pod \"network-metrics-daemon-4p7gl\" (UID: \"18353093-2e17-408e-b541-5dfb339343f0\") " pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:31.821151 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:31.821127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret\") pod \"global-pull-secret-syncer-w5wfm\" (UID: \"32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7\") " pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:31.821347 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:31.821202    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 11:51:31.821347 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:31.821223    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 11:51:31.821347 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:31.821262    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs podName:18353093-2e17-408e-b541-5dfb339343f0 nodeName:}" failed. No retries permitted until 2026-07-16 11:52:03.821247012 +0000 UTC m=+65.122024088 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs") pod "network-metrics-daemon-4p7gl" (UID: "18353093-2e17-408e-b541-5dfb339343f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 11:51:31.821347 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:31.821278    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret podName:32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7 nodeName:}" failed. No retries permitted until 2026-07-16 11:52:03.821270112 +0000 UTC m=+65.122047188 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret") pod "global-pull-secret-syncer-w5wfm" (UID: "32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7") : object "kube-system"/"original-pull-secret" not registered
Jul 16 11:51:31.921888 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:31.921851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4gmx\" (UniqueName: \"kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx\") pod \"network-check-target-8p6dq\" (UID: \"3768287e-bbac-495d-a015-f6fca7d1ec36\") " pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:31.922031 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:31.922012    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 11:51:31.922106 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:31.922052    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 11:51:31.922106 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:31.922066    2578 projected.go:196] Error preparing data for projected volume kube-api-access-x4gmx for pod openshift-network-diagnostics/network-check-target-8p6dq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 11:51:31.922207 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:31.922119    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx podName:3768287e-bbac-495d-a015-f6fca7d1ec36 nodeName:}" failed. No retries permitted until 2026-07-16 11:52:03.922106153 +0000 UTC m=+65.222883229 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x4gmx" (UniqueName: "kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx") pod "network-check-target-8p6dq" (UID: "3768287e-bbac-495d-a015-f6fca7d1ec36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 11:51:32.293752 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:32.293728    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:32.293913 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:32.293838    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5wfm" podUID="32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7"
Jul 16 11:51:32.293913 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:32.293885    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:32.294031 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:32.293958    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p6dq" podUID="3768287e-bbac-495d-a015-f6fca7d1ec36"
Jul 16 11:51:33.293409 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:33.293243    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:33.293562 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:33.293509    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p7gl" podUID="18353093-2e17-408e-b541-5dfb339343f0"
Jul 16 11:51:34.293733 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.293694    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:34.294202 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.293706    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:34.294202 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:34.293806    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p6dq" podUID="3768287e-bbac-495d-a015-f6fca7d1ec36"
Jul 16 11:51:34.294202 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:34.293909    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5wfm" podUID="32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7"
Jul 16 11:51:34.501703 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.501672    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeReady"
Jul 16 11:51:34.501839 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.501786    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 11:51:34.546231 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.546155    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qm4wx"]
Jul 16 11:51:34.548457 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.548397    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:51:34.549435 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.549196    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-crzvg"]
Jul 16 11:51:34.550729 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.550707    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-crzvg"
Jul 16 11:51:34.551135 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.550872    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 11:51:34.551135 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.550974    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 11:51:34.551135 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.551019    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b759k\""
Jul 16 11:51:34.552863 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.552842    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 11:51:34.553013 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.552995    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2lz8k\""
Jul 16 11:51:34.553102 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.553071    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 11:51:34.553283 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.553261    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 11:51:34.565982 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.565877    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qm4wx"]
Jul 16 11:51:34.568792 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.568699    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-crzvg"]
Jul 16 11:51:34.643256 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.643215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g68ms\" (UniqueName: \"kubernetes.io/projected/af3a84f5-1b60-40db-acd1-3624b84c09fa-kube-api-access-g68ms\") pod \"ingress-canary-crzvg\" (UID: \"af3a84f5-1b60-40db-acd1-3624b84c09fa\") " pod="openshift-ingress-canary/ingress-canary-crzvg"
Jul 16 11:51:34.643431 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.643266    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af3a84f5-1b60-40db-acd1-3624b84c09fa-cert\") pod \"ingress-canary-crzvg\" (UID: \"af3a84f5-1b60-40db-acd1-3624b84c09fa\") " pod="openshift-ingress-canary/ingress-canary-crzvg"
Jul 16 11:51:34.643431 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.643379    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls\") pod \"dns-default-qm4wx\" (UID: \"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a\") " pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:51:34.643431 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.643411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qwpd\" (UniqueName: \"kubernetes.io/projected/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-kube-api-access-8qwpd\") pod \"dns-default-qm4wx\" (UID: \"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a\") " pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:51:34.643599 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.643436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-config-volume\") pod \"dns-default-qm4wx\" (UID: \"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a\") " pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:51:34.643599 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.643544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-tmp-dir\") pod \"dns-default-qm4wx\" (UID: \"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a\") " pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:51:34.744595 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.744555    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-config-volume\") pod \"dns-default-qm4wx\" (UID: \"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a\") " pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:51:34.744777 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.744613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-tmp-dir\") pod \"dns-default-qm4wx\" (UID: \"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a\") " pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:51:34.744777 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.744648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g68ms\" (UniqueName: \"kubernetes.io/projected/af3a84f5-1b60-40db-acd1-3624b84c09fa-kube-api-access-g68ms\") pod \"ingress-canary-crzvg\" (UID: \"af3a84f5-1b60-40db-acd1-3624b84c09fa\") " pod="openshift-ingress-canary/ingress-canary-crzvg"
Jul 16 11:51:34.744777 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.744670    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af3a84f5-1b60-40db-acd1-3624b84c09fa-cert\") pod \"ingress-canary-crzvg\" (UID: \"af3a84f5-1b60-40db-acd1-3624b84c09fa\") " pod="openshift-ingress-canary/ingress-canary-crzvg"
Jul 16 11:51:34.744934 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.744786    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls\") pod \"dns-default-qm4wx\" (UID: \"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a\") " pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:51:34.744934 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.744853    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qwpd\" (UniqueName: \"kubernetes.io/projected/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-kube-api-access-8qwpd\") pod \"dns-default-qm4wx\" (UID: \"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a\") " pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:51:34.745021 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:34.744929    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 11:51:34.745021 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:34.744999    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls podName:2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a nodeName:}" failed. No retries permitted until 2026-07-16 11:51:35.244976975 +0000 UTC m=+36.545754054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls") pod "dns-default-qm4wx" (UID: "2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a") : secret "dns-default-metrics-tls" not found
Jul 16 11:51:34.745275 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.745252    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-tmp-dir\") pod \"dns-default-qm4wx\" (UID: \"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a\") " pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:51:34.745344 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.745310    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-config-volume\") pod \"dns-default-qm4wx\" (UID: \"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a\") " pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:51:34.749185 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.749155    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af3a84f5-1b60-40db-acd1-3624b84c09fa-cert\") pod \"ingress-canary-crzvg\" (UID: \"af3a84f5-1b60-40db-acd1-3624b84c09fa\") " pod="openshift-ingress-canary/ingress-canary-crzvg"
Jul 16 11:51:34.753505 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.753480    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qwpd\" (UniqueName: \"kubernetes.io/projected/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-kube-api-access-8qwpd\") pod \"dns-default-qm4wx\" (UID: \"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a\") " pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:51:34.753626 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.753500    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g68ms\" (UniqueName: \"kubernetes.io/projected/af3a84f5-1b60-40db-acd1-3624b84c09fa-kube-api-access-g68ms\") pod \"ingress-canary-crzvg\" (UID: \"af3a84f5-1b60-40db-acd1-3624b84c09fa\") " pod="openshift-ingress-canary/ingress-canary-crzvg"
Jul 16 11:51:34.869922 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:34.869842    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-crzvg"
Jul 16 11:51:35.017142 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:35.017112    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-crzvg"]
Jul 16 11:51:35.021769 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:51:35.021741    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3a84f5_1b60_40db_acd1_3624b84c09fa.slice/crio-114d157b5d81f9ee6f1cc377e7b1a25ec7835eed7ee4b9fb4e1d67c611b71672 WatchSource:0}: Error finding container 114d157b5d81f9ee6f1cc377e7b1a25ec7835eed7ee4b9fb4e1d67c611b71672: Status 404 returned error can't find the container with id 114d157b5d81f9ee6f1cc377e7b1a25ec7835eed7ee4b9fb4e1d67c611b71672
Jul 16 11:51:35.248726 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:35.248648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls\") pod \"dns-default-qm4wx\" (UID: \"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a\") " pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:51:35.248876 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:35.248821    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 11:51:35.248926 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:35.248900    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls podName:2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a nodeName:}" failed. No retries permitted until 2026-07-16 11:51:36.248879081 +0000 UTC m=+37.549656160 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls") pod "dns-default-qm4wx" (UID: "2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a") : secret "dns-default-metrics-tls" not found
Jul 16 11:51:35.293603 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:35.293562    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:51:35.296083 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:35.296001    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 11:51:35.296083 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:35.296055    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vn7fb\""
Jul 16 11:51:35.496867 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:35.496830    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-crzvg" event={"ID":"af3a84f5-1b60-40db-acd1-3624b84c09fa","Type":"ContainerStarted","Data":"114d157b5d81f9ee6f1cc377e7b1a25ec7835eed7ee4b9fb4e1d67c611b71672"}
Jul 16 11:51:36.257907 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:36.257866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls\") pod \"dns-default-qm4wx\" (UID: \"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a\") " pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:51:36.258121 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:36.258005    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 11:51:36.258121 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:36.258100    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls podName:2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a nodeName:}" failed. No retries permitted until 2026-07-16 11:51:38.258078258 +0000 UTC m=+39.558855336 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls") pod "dns-default-qm4wx" (UID: "2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a") : secret "dns-default-metrics-tls" not found
Jul 16 11:51:36.294098 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:36.294063    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:51:36.294273 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:36.294246    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:51:36.296473 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:36.296450    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 11:51:36.296824 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:36.296451    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gtptt\""
Jul 16 11:51:36.297298 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:36.297241    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 11:51:36.297298 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:36.297283    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 11:51:38.271191 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:38.271155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls\") pod \"dns-default-qm4wx\" (UID: \"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a\") " pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:51:38.271632 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:38.271304    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 11:51:38.271632 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:38.271367    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls podName:2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a nodeName:}" failed. No retries permitted until 2026-07-16 11:51:42.27135138 +0000 UTC m=+43.572128456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls") pod "dns-default-qm4wx" (UID: "2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a") : secret "dns-default-metrics-tls" not found
Jul 16 11:51:38.506295 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:38.506266    2578 generic.go:356] "Generic (PLEG): container finished" podID="4c9ae01e-f7f2-4296-a0b0-600bd31f36a8" containerID="46099eac6211ca716481e3039073859fd8187531879846c15073a63040544435" exitCode=0
Jul 16 11:51:38.506406 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:38.506346    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgflz" event={"ID":"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8","Type":"ContainerDied","Data":"46099eac6211ca716481e3039073859fd8187531879846c15073a63040544435"}
Jul 16 11:51:38.510117 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:38.510091    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-crzvg" event={"ID":"af3a84f5-1b60-40db-acd1-3624b84c09fa","Type":"ContainerStarted","Data":"f9821b2223bbbc4d14993990d6bd7ae56a058744c62eb0ab934827ba28ea40d5"}
Jul 16 11:51:38.559701 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:38.559663    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-crzvg" podStartSLOduration=1.3027217150000001 podStartE2EDuration="4.559647418s" totalImagesPullingTime="3.256925703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:51:38.559109843 +0000 UTC m=+39.859886944" watchObservedRunningTime="2026-07-16 11:51:38.559647418 +0000 UTC m=+39.860424517"
Jul 16 11:51:39.515250 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:39.515219    2578 generic.go:356] "Generic (PLEG): container finished" podID="4c9ae01e-f7f2-4296-a0b0-600bd31f36a8" containerID="d9311e3748f2f1b3bf94f0561e7afefa02af2a48548a467fbc8a5615655b3055" exitCode=0
Jul 16 11:51:39.515650 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:39.515301    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgflz" event={"ID":"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8","Type":"ContainerDied","Data":"d9311e3748f2f1b3bf94f0561e7afefa02af2a48548a467fbc8a5615655b3055"}
Jul 16 11:51:40.521183 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:40.520943    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgflz" event={"ID":"4c9ae01e-f7f2-4296-a0b0-600bd31f36a8","Type":"ContainerStarted","Data":"b783d8865245f23b24a52a722cc3600f1f0f3522a858e1a1265fe89f8716e81e"}
Jul 16 11:51:40.549758 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:40.549711    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lgflz" podStartSLOduration=6.104865338 podStartE2EDuration="40.549698116s" totalImagesPullingTime="34.444832778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:51:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:51:40.546168652 +0000 UTC m=+41.846945750" watchObservedRunningTime="2026-07-16 11:51:40.549698116 +0000 UTC m=+41.850475215"
Jul 16 11:51:42.297096 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.297058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls\") pod \"dns-default-qm4wx\" (UID: \"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a\") " pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:51:42.297442 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:42.297172    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 11:51:42.297442 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:42.297226    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls podName:2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a nodeName:}" failed. No retries permitted until 2026-07-16 11:51:50.297212478 +0000 UTC m=+51.597989554 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls") pod "dns-default-qm4wx" (UID: "2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a") : secret "dns-default-metrics-tls" not found
Jul 16 11:51:42.762576 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.762543    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq"]
Jul 16 11:51:42.795074 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.795029    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-qmc55"]
Jul 16 11:51:42.795204 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.795145    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq"
Jul 16 11:51:42.797641 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.797621    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 11:51:42.798905 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.798881    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 16 11:51:42.799061 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.798931    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 11:51:42.799061 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.798993    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-65t59\""
Jul 16 11:51:42.799198 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.799121    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 16 11:51:42.818923 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.818896    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-85b9cb9c46-gmf4x"]
Jul 16 11:51:42.819078 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.818983    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-qmc55"
Jul 16 11:51:42.821121 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.821101    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 16 11:51:42.821223 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.821132    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 16 11:51:42.821223 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.821138    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 11:51:42.821412 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.821397    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ccsx7\""
Jul 16 11:51:42.821454 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.821447    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 11:51:42.827508 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.827484    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 16 11:51:42.835493 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.835476    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq"]
Jul 16 11:51:42.835583 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.835501    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-qmc55"]
Jul 16 11:51:42.835583 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.835515    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-85b9cb9c46-gmf4x"]
Jul 16 11:51:42.835647 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.835583    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:42.837773 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.837747    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 16 11:51:42.837865 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.837821    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vct7t\""
Jul 16 11:51:42.837865 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.837829    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 16 11:51:42.837865 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.837851    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 16 11:51:42.838142 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.838121    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 16 11:51:42.838251 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.838155    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 16 11:51:42.838251 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.838121    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 16 11:51:42.900919 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.900890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/64f7c050-5fde-47f1-9e42-5deaca93acd7-snapshots\") pod \"insights-operator-68d9dd94cf-qmc55\" (UID: \"64f7c050-5fde-47f1-9e42-5deaca93acd7\") " pod="openshift-insights/insights-operator-68d9dd94cf-qmc55"
Jul 16 11:51:42.901089 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.900932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-vpthq\" (UID: \"715b8182-f04a-4e70-9766-1f4075a48dab\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq"
Jul 16 11:51:42.901089 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.900959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64f7c050-5fde-47f1-9e42-5deaca93acd7-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-qmc55\" (UID: \"64f7c050-5fde-47f1-9e42-5deaca93acd7\") " pod="openshift-insights/insights-operator-68d9dd94cf-qmc55"
Jul 16 11:51:42.901089 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.900974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64f7c050-5fde-47f1-9e42-5deaca93acd7-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-qmc55\" (UID: \"64f7c050-5fde-47f1-9e42-5deaca93acd7\") " pod="openshift-insights/insights-operator-68d9dd94cf-qmc55"
Jul 16 11:51:42.901089 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.901021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64f7c050-5fde-47f1-9e42-5deaca93acd7-serving-cert\") pod \"insights-operator-68d9dd94cf-qmc55\" (UID: \"64f7c050-5fde-47f1-9e42-5deaca93acd7\") " pod="openshift-insights/insights-operator-68d9dd94cf-qmc55"
Jul 16 11:51:42.901089 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.901075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckj5r\" (UniqueName: \"kubernetes.io/projected/64f7c050-5fde-47f1-9e42-5deaca93acd7-kube-api-access-ckj5r\") pod \"insights-operator-68d9dd94cf-qmc55\" (UID: \"64f7c050-5fde-47f1-9e42-5deaca93acd7\") " pod="openshift-insights/insights-operator-68d9dd94cf-qmc55"
Jul 16 11:51:42.901331 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.901140    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f82dc\" (UniqueName: \"kubernetes.io/projected/715b8182-f04a-4e70-9766-1f4075a48dab-kube-api-access-f82dc\") pod \"cluster-monitoring-operator-5f7f494dd4-vpthq\" (UID: \"715b8182-f04a-4e70-9766-1f4075a48dab\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq"
Jul 16 11:51:42.901331 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.901179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/715b8182-f04a-4e70-9766-1f4075a48dab-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-vpthq\" (UID: \"715b8182-f04a-4e70-9766-1f4075a48dab\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq"
Jul 16 11:51:42.901331 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:42.901236    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64f7c050-5fde-47f1-9e42-5deaca93acd7-tmp\") pod \"insights-operator-68d9dd94cf-qmc55\" (UID: \"64f7c050-5fde-47f1-9e42-5deaca93acd7\") " pod="openshift-insights/insights-operator-68d9dd94cf-qmc55"
Jul 16 11:51:43.002240 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.002210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f82dc\" (UniqueName: \"kubernetes.io/projected/715b8182-f04a-4e70-9766-1f4075a48dab-kube-api-access-f82dc\") pod \"cluster-monitoring-operator-5f7f494dd4-vpthq\" (UID: \"715b8182-f04a-4e70-9766-1f4075a48dab\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq"
Jul 16 11:51:43.002364 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.002249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5828g\" (UniqueName: \"kubernetes.io/projected/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-kube-api-access-5828g\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:43.002364 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.002279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/715b8182-f04a-4e70-9766-1f4075a48dab-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-vpthq\" (UID: \"715b8182-f04a-4e70-9766-1f4075a48dab\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq"
Jul 16 11:51:43.002364 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.002333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64f7c050-5fde-47f1-9e42-5deaca93acd7-tmp\") pod \"insights-operator-68d9dd94cf-qmc55\" (UID: \"64f7c050-5fde-47f1-9e42-5deaca93acd7\") " pod="openshift-insights/insights-operator-68d9dd94cf-qmc55"
Jul 16 11:51:43.002538 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.002370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:43.002538 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.002391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-stats-auth\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:43.002538 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.002432    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/64f7c050-5fde-47f1-9e42-5deaca93acd7-snapshots\") pod \"insights-operator-68d9dd94cf-qmc55\" (UID: \"64f7c050-5fde-47f1-9e42-5deaca93acd7\") " pod="openshift-insights/insights-operator-68d9dd94cf-qmc55"
Jul 16 11:51:43.002538 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.002477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:43.002538 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.002527    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-vpthq\" (UID: \"715b8182-f04a-4e70-9766-1f4075a48dab\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq"
Jul 16 11:51:43.002778 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.002556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64f7c050-5fde-47f1-9e42-5deaca93acd7-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-qmc55\" (UID: \"64f7c050-5fde-47f1-9e42-5deaca93acd7\") " pod="openshift-insights/insights-operator-68d9dd94cf-qmc55"
Jul 16 11:51:43.002778 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.002578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64f7c050-5fde-47f1-9e42-5deaca93acd7-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-qmc55\" (UID: \"64f7c050-5fde-47f1-9e42-5deaca93acd7\") " pod="openshift-insights/insights-operator-68d9dd94cf-qmc55"
Jul 16 11:51:43.002778 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.002601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64f7c050-5fde-47f1-9e42-5deaca93acd7-serving-cert\") pod \"insights-operator-68d9dd94cf-qmc55\" (UID: \"64f7c050-5fde-47f1-9e42-5deaca93acd7\") " pod="openshift-insights/insights-operator-68d9dd94cf-qmc55"
Jul 16 11:51:43.002778 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.002625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckj5r\" (UniqueName: \"kubernetes.io/projected/64f7c050-5fde-47f1-9e42-5deaca93acd7-kube-api-access-ckj5r\") pod \"insights-operator-68d9dd94cf-qmc55\" (UID: \"64f7c050-5fde-47f1-9e42-5deaca93acd7\") " pod="openshift-insights/insights-operator-68d9dd94cf-qmc55"
Jul 16 11:51:43.002778 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:43.002658    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 11:51:43.002778 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:43.002723    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls podName:715b8182-f04a-4e70-9766-1f4075a48dab nodeName:}" failed. No retries permitted until 2026-07-16 11:51:43.50270479 +0000 UTC m=+44.803481892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-vpthq" (UID: "715b8182-f04a-4e70-9766-1f4075a48dab") : secret "cluster-monitoring-operator-tls" not found
Jul 16 11:51:43.003079 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.002664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-default-certificate\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:43.003079 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.002954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64f7c050-5fde-47f1-9e42-5deaca93acd7-tmp\") pod \"insights-operator-68d9dd94cf-qmc55\" (UID: \"64f7c050-5fde-47f1-9e42-5deaca93acd7\") " pod="openshift-insights/insights-operator-68d9dd94cf-qmc55"
Jul 16 11:51:43.003079 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.003028    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/64f7c050-5fde-47f1-9e42-5deaca93acd7-snapshots\") pod \"insights-operator-68d9dd94cf-qmc55\" (UID: \"64f7c050-5fde-47f1-9e42-5deaca93acd7\") " pod="openshift-insights/insights-operator-68d9dd94cf-qmc55"
Jul 16 11:51:43.003244 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.003199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/715b8182-f04a-4e70-9766-1f4075a48dab-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-vpthq\" (UID: \"715b8182-f04a-4e70-9766-1f4075a48dab\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq"
Jul 16 11:51:43.003281 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.003240    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64f7c050-5fde-47f1-9e42-5deaca93acd7-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-qmc55\" (UID: \"64f7c050-5fde-47f1-9e42-5deaca93acd7\") " pod="openshift-insights/insights-operator-68d9dd94cf-qmc55"
Jul 16 11:51:43.003654 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.003633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64f7c050-5fde-47f1-9e42-5deaca93acd7-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-qmc55\" (UID: \"64f7c050-5fde-47f1-9e42-5deaca93acd7\") " pod="openshift-insights/insights-operator-68d9dd94cf-qmc55"
Jul 16 11:51:43.006510 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.006494    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64f7c050-5fde-47f1-9e42-5deaca93acd7-serving-cert\") pod \"insights-operator-68d9dd94cf-qmc55\" (UID: \"64f7c050-5fde-47f1-9e42-5deaca93acd7\") " pod="openshift-insights/insights-operator-68d9dd94cf-qmc55"
Jul 16 11:51:43.017264 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.017218    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckj5r\" (UniqueName: \"kubernetes.io/projected/64f7c050-5fde-47f1-9e42-5deaca93acd7-kube-api-access-ckj5r\") pod \"insights-operator-68d9dd94cf-qmc55\" (UID: \"64f7c050-5fde-47f1-9e42-5deaca93acd7\") " pod="openshift-insights/insights-operator-68d9dd94cf-qmc55"
Jul 16 11:51:43.017345 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.017262    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f82dc\" (UniqueName: \"kubernetes.io/projected/715b8182-f04a-4e70-9766-1f4075a48dab-kube-api-access-f82dc\") pod \"cluster-monitoring-operator-5f7f494dd4-vpthq\" (UID: \"715b8182-f04a-4e70-9766-1f4075a48dab\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq"
Jul 16 11:51:43.103645 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.103613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5828g\" (UniqueName: \"kubernetes.io/projected/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-kube-api-access-5828g\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:43.103778 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.103660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:43.103778 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.103676    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-stats-auth\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:43.103778 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.103704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:43.103937 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:43.103792    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 11:51:43.103937 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.103811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-default-certificate\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:43.103937 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:43.103845    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle podName:f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e nodeName:}" failed. No retries permitted until 2026-07-16 11:51:43.603826469 +0000 UTC m=+44.904603570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle") pod "router-default-85b9cb9c46-gmf4x" (UID: "f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e") : configmap references non-existent config key: service-ca.crt
Jul 16 11:51:43.103937 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:43.103890    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs podName:f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e nodeName:}" failed. No retries permitted until 2026-07-16 11:51:43.603873509 +0000 UTC m=+44.904650588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs") pod "router-default-85b9cb9c46-gmf4x" (UID: "f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e") : secret "router-metrics-certs-default" not found
Jul 16 11:51:43.105936 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.105911    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-stats-auth\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:43.106010 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.105954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-default-certificate\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:43.112130 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.112111    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5828g\" (UniqueName: \"kubernetes.io/projected/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-kube-api-access-5828g\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:43.126983 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.126961    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-qmc55"
Jul 16 11:51:43.277363 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.277296    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-qmc55"]
Jul 16 11:51:43.281557 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:51:43.281536    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f7c050_5fde_47f1_9e42_5deaca93acd7.slice/crio-eac2ee43cd206ddec44772b71cefae9e3de1ddfacf5f0fcb368ae1e8a4991cbb WatchSource:0}: Error finding container eac2ee43cd206ddec44772b71cefae9e3de1ddfacf5f0fcb368ae1e8a4991cbb: Status 404 returned error can't find the container with id eac2ee43cd206ddec44772b71cefae9e3de1ddfacf5f0fcb368ae1e8a4991cbb
Jul 16 11:51:43.506742 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.506704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-vpthq\" (UID: \"715b8182-f04a-4e70-9766-1f4075a48dab\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq"
Jul 16 11:51:43.507158 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:43.506847    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 11:51:43.507158 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:43.506910    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls podName:715b8182-f04a-4e70-9766-1f4075a48dab nodeName:}" failed. No retries permitted until 2026-07-16 11:51:44.5068954 +0000 UTC m=+45.807672476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-vpthq" (UID: "715b8182-f04a-4e70-9766-1f4075a48dab") : secret "cluster-monitoring-operator-tls" not found
Jul 16 11:51:43.527663 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.527596    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-qmc55" event={"ID":"64f7c050-5fde-47f1-9e42-5deaca93acd7","Type":"ContainerStarted","Data":"eac2ee43cd206ddec44772b71cefae9e3de1ddfacf5f0fcb368ae1e8a4991cbb"}
Jul 16 11:51:43.608068 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.608023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:43.608190 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:43.608082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:43.608240 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:43.608210    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle podName:f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e nodeName:}" failed. No retries permitted until 2026-07-16 11:51:44.608186917 +0000 UTC m=+45.908963996 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle") pod "router-default-85b9cb9c46-gmf4x" (UID: "f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e") : configmap references non-existent config key: service-ca.crt
Jul 16 11:51:43.608283 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:43.608235    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 11:51:43.608313 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:43.608282    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs podName:f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e nodeName:}" failed. No retries permitted until 2026-07-16 11:51:44.608270013 +0000 UTC m=+45.909047095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs") pod "router-default-85b9cb9c46-gmf4x" (UID: "f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e") : secret "router-metrics-certs-default" not found
Jul 16 11:51:44.514825 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:44.514783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-vpthq\" (UID: \"715b8182-f04a-4e70-9766-1f4075a48dab\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq"
Jul 16 11:51:44.515275 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:44.514934    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 11:51:44.515275 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:44.514998    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls podName:715b8182-f04a-4e70-9766-1f4075a48dab nodeName:}" failed. No retries permitted until 2026-07-16 11:51:46.514983011 +0000 UTC m=+47.815760087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-vpthq" (UID: "715b8182-f04a-4e70-9766-1f4075a48dab") : secret "cluster-monitoring-operator-tls" not found
Jul 16 11:51:44.615542 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:44.615506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:44.615691 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:44.615556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:44.615728 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:44.615693    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 11:51:44.615728 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:44.615693    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle podName:f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e nodeName:}" failed. No retries permitted until 2026-07-16 11:51:46.615674598 +0000 UTC m=+47.916451688 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle") pod "router-default-85b9cb9c46-gmf4x" (UID: "f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e") : configmap references non-existent config key: service-ca.crt
Jul 16 11:51:44.615823 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:44.615741    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs podName:f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e nodeName:}" failed. No retries permitted until 2026-07-16 11:51:46.615729388 +0000 UTC m=+47.916506469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs") pod "router-default-85b9cb9c46-gmf4x" (UID: "f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e") : secret "router-metrics-certs-default" not found
Jul 16 11:51:45.660768 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.660730    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-z5cnp"]
Jul 16 11:51:45.684995 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.684964    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-z5cnp"]
Jul 16 11:51:45.684995 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.684999    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb"]
Jul 16 11:51:45.685177 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.685090    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-z5cnp"
Jul 16 11:51:45.687430 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.687397    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 16 11:51:45.687606 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.687451    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 16 11:51:45.688306 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.688288    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-8v998\""
Jul 16 11:51:45.699255 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.699237    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb"]
Jul 16 11:51:45.699336 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.699323    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb"
Jul 16 11:51:45.701710 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.701686    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 16 11:51:45.701826 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.701724    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 16 11:51:45.702220 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.702206    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cvcg7\""
Jul 16 11:51:45.702297 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.702268    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 16 11:51:45.825339 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.825312    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkzf5\" (UniqueName: \"kubernetes.io/projected/ce18ca5b-8314-4db8-9fc6-7717fcb0b0cc-kube-api-access-qkzf5\") pod \"volume-data-source-validator-78495597f6-z5cnp\" (UID: \"ce18ca5b-8314-4db8-9fc6-7717fcb0b0cc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-z5cnp"
Jul 16 11:51:45.825451 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.825354    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9kmwb\" (UID: \"4391689c-b45b-42f6-9e95-b7ea48a4e131\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb"
Jul 16 11:51:45.825451 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.825379    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sbm6\" (UniqueName: \"kubernetes.io/projected/4391689c-b45b-42f6-9e95-b7ea48a4e131-kube-api-access-9sbm6\") pod \"cluster-samples-operator-65db858894-9kmwb\" (UID: \"4391689c-b45b-42f6-9e95-b7ea48a4e131\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb"
Jul 16 11:51:45.926343 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.926251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkzf5\" (UniqueName: \"kubernetes.io/projected/ce18ca5b-8314-4db8-9fc6-7717fcb0b0cc-kube-api-access-qkzf5\") pod \"volume-data-source-validator-78495597f6-z5cnp\" (UID: \"ce18ca5b-8314-4db8-9fc6-7717fcb0b0cc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-z5cnp"
Jul 16 11:51:45.926343 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.926305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9kmwb\" (UID: \"4391689c-b45b-42f6-9e95-b7ea48a4e131\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb"
Jul 16 11:51:45.926343 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.926342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sbm6\" (UniqueName: \"kubernetes.io/projected/4391689c-b45b-42f6-9e95-b7ea48a4e131-kube-api-access-9sbm6\") pod \"cluster-samples-operator-65db858894-9kmwb\" (UID: \"4391689c-b45b-42f6-9e95-b7ea48a4e131\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb"
Jul 16 11:51:45.926606 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:45.926411    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 11:51:45.926606 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:45.926479    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls podName:4391689c-b45b-42f6-9e95-b7ea48a4e131 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:46.426460717 +0000 UTC m=+47.727237797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls") pod "cluster-samples-operator-65db858894-9kmwb" (UID: "4391689c-b45b-42f6-9e95-b7ea48a4e131") : secret "samples-operator-tls" not found
Jul 16 11:51:45.939278 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.939247    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkzf5\" (UniqueName: \"kubernetes.io/projected/ce18ca5b-8314-4db8-9fc6-7717fcb0b0cc-kube-api-access-qkzf5\") pod \"volume-data-source-validator-78495597f6-z5cnp\" (UID: \"ce18ca5b-8314-4db8-9fc6-7717fcb0b0cc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-z5cnp"
Jul 16 11:51:45.939410 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.939379    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sbm6\" (UniqueName: \"kubernetes.io/projected/4391689c-b45b-42f6-9e95-b7ea48a4e131-kube-api-access-9sbm6\") pod \"cluster-samples-operator-65db858894-9kmwb\" (UID: \"4391689c-b45b-42f6-9e95-b7ea48a4e131\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb"
Jul 16 11:51:45.994208 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:45.994176    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-z5cnp"
Jul 16 11:51:46.138496 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.138462    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-z5cnp"]
Jul 16 11:51:46.142192 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:51:46.142157    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce18ca5b_8314_4db8_9fc6_7717fcb0b0cc.slice/crio-5e024d6107fe3a8361dbaa167d9743b41857e63e2b76b8d2b6ee200688a73035 WatchSource:0}: Error finding container 5e024d6107fe3a8361dbaa167d9743b41857e63e2b76b8d2b6ee200688a73035: Status 404 returned error can't find the container with id 5e024d6107fe3a8361dbaa167d9743b41857e63e2b76b8d2b6ee200688a73035
Jul 16 11:51:46.430515 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.430488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9kmwb\" (UID: \"4391689c-b45b-42f6-9e95-b7ea48a4e131\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb"
Jul 16 11:51:46.430699 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:46.430599    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 11:51:46.430699 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:46.430653    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls podName:4391689c-b45b-42f6-9e95-b7ea48a4e131 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:47.430640451 +0000 UTC m=+48.731417527 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls") pod "cluster-samples-operator-65db858894-9kmwb" (UID: "4391689c-b45b-42f6-9e95-b7ea48a4e131") : secret "samples-operator-tls" not found
Jul 16 11:51:46.531434 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.531390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-vpthq\" (UID: \"715b8182-f04a-4e70-9766-1f4075a48dab\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq"
Jul 16 11:51:46.531586 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:46.531563    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 11:51:46.531657 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:46.531647    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls podName:715b8182-f04a-4e70-9766-1f4075a48dab nodeName:}" failed. No retries permitted until 2026-07-16 11:51:50.531626516 +0000 UTC m=+51.832403595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-vpthq" (UID: "715b8182-f04a-4e70-9766-1f4075a48dab") : secret "cluster-monitoring-operator-tls" not found
Jul 16 11:51:46.536675 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.536649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-qmc55" event={"ID":"64f7c050-5fde-47f1-9e42-5deaca93acd7","Type":"ContainerStarted","Data":"1086c4da2c07cead574fdc97ec94c5c6cf3b5029953768fe356c3fc66cb55cf0"}
Jul 16 11:51:46.537773 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.537754    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-z5cnp" event={"ID":"ce18ca5b-8314-4db8-9fc6-7717fcb0b0cc","Type":"ContainerStarted","Data":"5e024d6107fe3a8361dbaa167d9743b41857e63e2b76b8d2b6ee200688a73035"}
Jul 16 11:51:46.572695 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.572643    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-qmc55" podStartSLOduration=2.07743074 podStartE2EDuration="4.572625552s" totalImagesPullingTime="2.495194812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:51:46.571761938 +0000 UTC m=+47.872539049" watchObservedRunningTime="2026-07-16 11:51:46.572625552 +0000 UTC m=+47.873402652"
Jul 16 11:51:46.632681 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.632647    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:46.632831 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.632713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:46.632893 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:46.632838    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle podName:f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e nodeName:}" failed. No retries permitted until 2026-07-16 11:51:50.632812547 +0000 UTC m=+51.933589643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle") pod "router-default-85b9cb9c46-gmf4x" (UID: "f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e") : configmap references non-existent config key: service-ca.crt
Jul 16 11:51:46.633167 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:46.633148    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 11:51:46.633268 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:46.633212    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs podName:f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e nodeName:}" failed. No retries permitted until 2026-07-16 11:51:50.633195406 +0000 UTC m=+51.933972496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs") pod "router-default-85b9cb9c46-gmf4x" (UID: "f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e") : secret "router-metrics-certs-default" not found
Jul 16 11:51:46.671995 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.671969    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rr62s"]
Jul 16 11:51:46.695813 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.695763    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:46.698961 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.698943    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 11:51:46.699148 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.699129    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qlctg\""
Jul 16 11:51:46.700654 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.700637    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 11:51:46.703292 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.703272    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rr62s"]
Jul 16 11:51:46.834880 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.834851    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:46.835017 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.834894    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81f767f1-2e6f-4ad1-b92d-612c527a97b3-data-volume\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:46.835017 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.834922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81f767f1-2e6f-4ad1-b92d-612c527a97b3-crio-socket\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:46.835017 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.834954    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k95k7\" (UniqueName: \"kubernetes.io/projected/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-api-access-k95k7\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:46.835017 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.835001    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:46.935360 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.935330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81f767f1-2e6f-4ad1-b92d-612c527a97b3-crio-socket\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:46.935494 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.935391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k95k7\" (UniqueName: \"kubernetes.io/projected/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-api-access-k95k7\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:46.935494 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.935474    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:46.935617 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.935532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:46.935617 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.935547    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81f767f1-2e6f-4ad1-b92d-612c527a97b3-crio-socket\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:46.935617 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.935581    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81f767f1-2e6f-4ad1-b92d-612c527a97b3-data-volume\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:46.935617 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:46.935595    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 11:51:46.935803 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:46.935669    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-rbac-proxy-cm podName:81f767f1-2e6f-4ad1-b92d-612c527a97b3 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:47.435648726 +0000 UTC m=+48.736425817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rr62s" (UID: "81f767f1-2e6f-4ad1-b92d-612c527a97b3") : configmap "kube-rbac-proxy" not found
Jul 16 11:51:46.935803 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:46.935685    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 11:51:46.935803 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:46.935731    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls podName:81f767f1-2e6f-4ad1-b92d-612c527a97b3 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:47.435718665 +0000 UTC m=+48.736495749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rr62s" (UID: "81f767f1-2e6f-4ad1-b92d-612c527a97b3") : secret "insights-runtime-extractor-tls" not found
Jul 16 11:51:46.935967 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.935948    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81f767f1-2e6f-4ad1-b92d-612c527a97b3-data-volume\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:46.944187 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:46.944159    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k95k7\" (UniqueName: \"kubernetes.io/projected/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-api-access-k95k7\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:47.441261 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:47.441228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9kmwb\" (UID: \"4391689c-b45b-42f6-9e95-b7ea48a4e131\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb"
Jul 16 11:51:47.441412 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:47.441276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:47.441412 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:47.441379    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 11:51:47.441484 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:47.441442    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 11:51:47.441484 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:47.441382    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:47.441484 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:47.441473    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 11:51:47.441569 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:47.441445    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls podName:4391689c-b45b-42f6-9e95-b7ea48a4e131 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:49.441426372 +0000 UTC m=+50.742203448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls") pod "cluster-samples-operator-65db858894-9kmwb" (UID: "4391689c-b45b-42f6-9e95-b7ea48a4e131") : secret "samples-operator-tls" not found
Jul 16 11:51:47.441569 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:47.441534    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls podName:81f767f1-2e6f-4ad1-b92d-612c527a97b3 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:48.44152058 +0000 UTC m=+49.742297661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rr62s" (UID: "81f767f1-2e6f-4ad1-b92d-612c527a97b3") : secret "insights-runtime-extractor-tls" not found
Jul 16 11:51:47.441652 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:47.441570    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-rbac-proxy-cm podName:81f767f1-2e6f-4ad1-b92d-612c527a97b3 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:48.441561558 +0000 UTC m=+49.742338634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rr62s" (UID: "81f767f1-2e6f-4ad1-b92d-612c527a97b3") : configmap "kube-rbac-proxy" not found
Jul 16 11:51:48.135502 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:48.135480    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4sj9h_8b123f16-31a0-49be-906c-017bdc2602e6/dns-node-resolver/0.log"
Jul 16 11:51:48.449311 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:48.449274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:48.449475 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:48.449326    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:48.449475 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:48.449401    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 11:51:48.449475 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:48.449466    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 11:51:48.449569 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:48.449470    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-rbac-proxy-cm podName:81f767f1-2e6f-4ad1-b92d-612c527a97b3 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:50.449454268 +0000 UTC m=+51.750231347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rr62s" (UID: "81f767f1-2e6f-4ad1-b92d-612c527a97b3") : configmap "kube-rbac-proxy" not found
Jul 16 11:51:48.449569 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:48.449515    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls podName:81f767f1-2e6f-4ad1-b92d-612c527a97b3 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:50.449503054 +0000 UTC m=+51.750280131 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rr62s" (UID: "81f767f1-2e6f-4ad1-b92d-612c527a97b3") : secret "insights-runtime-extractor-tls" not found
Jul 16 11:51:48.545201 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:48.545172    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-z5cnp" event={"ID":"ce18ca5b-8314-4db8-9fc6-7717fcb0b0cc","Type":"ContainerStarted","Data":"2823d1fb4aec7c07ce39394219e806f424bcc0466159023aad7020a5f84a5284"}
Jul 16 11:51:48.580878 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:48.580813    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-z5cnp" podStartSLOduration=1.704771408 podStartE2EDuration="3.580787798s" totalImagesPullingTime="1.87601639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:51:48.578946556 +0000 UTC m=+49.879723656" watchObservedRunningTime="2026-07-16 11:51:48.580787798 +0000 UTC m=+49.881564897"
Jul 16 11:51:49.335867 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:49.335839    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vlbks_c5d99cb5-959d-4a75-89e7-fa68635e416d/node-ca/0.log"
Jul 16 11:51:49.457070 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:49.457025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9kmwb\" (UID: \"4391689c-b45b-42f6-9e95-b7ea48a4e131\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb"
Jul 16 11:51:49.457239 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:49.457173    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 11:51:49.457290 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:49.457244    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls podName:4391689c-b45b-42f6-9e95-b7ea48a4e131 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:53.45722401 +0000 UTC m=+54.758001103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls") pod "cluster-samples-operator-65db858894-9kmwb" (UID: "4391689c-b45b-42f6-9e95-b7ea48a4e131") : secret "samples-operator-tls" not found
Jul 16 11:51:49.736276 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:49.736201    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-crzvg_af3a84f5-1b60-40db-acd1-3624b84c09fa/serve-healthcheck-canary/0.log"
Jul 16 11:51:50.364082 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:50.364029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls\") pod \"dns-default-qm4wx\" (UID: \"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a\") " pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:51:50.364469 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:50.364176    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 11:51:50.364469 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:50.364237    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls podName:2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a nodeName:}" failed. No retries permitted until 2026-07-16 11:52:06.364222853 +0000 UTC m=+67.664999928 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls") pod "dns-default-qm4wx" (UID: "2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a") : secret "dns-default-metrics-tls" not found
Jul 16 11:51:50.464615 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:50.464582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:50.464781 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:50.464689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:50.464781 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:50.464702    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 11:51:50.464781 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:50.464755    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 11:51:50.464931 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:50.464792    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls podName:81f767f1-2e6f-4ad1-b92d-612c527a97b3 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:54.464768748 +0000 UTC m=+55.765545826 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rr62s" (UID: "81f767f1-2e6f-4ad1-b92d-612c527a97b3") : secret "insights-runtime-extractor-tls" not found
Jul 16 11:51:50.464931 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:50.464812    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-rbac-proxy-cm podName:81f767f1-2e6f-4ad1-b92d-612c527a97b3 nodeName:}" failed. No retries permitted until 2026-07-16 11:51:54.464801288 +0000 UTC m=+55.765578367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rr62s" (UID: "81f767f1-2e6f-4ad1-b92d-612c527a97b3") : configmap "kube-rbac-proxy" not found
Jul 16 11:51:50.565187 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:50.565162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-vpthq\" (UID: \"715b8182-f04a-4e70-9766-1f4075a48dab\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq"
Jul 16 11:51:50.565294 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:50.565271    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 11:51:50.565332 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:50.565312    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls podName:715b8182-f04a-4e70-9766-1f4075a48dab nodeName:}" failed. No retries permitted until 2026-07-16 11:51:58.565301618 +0000 UTC m=+59.866078694 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-vpthq" (UID: "715b8182-f04a-4e70-9766-1f4075a48dab") : secret "cluster-monitoring-operator-tls" not found
Jul 16 11:51:50.665725 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:50.665664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:50.665825 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:50.665751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:50.665825 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:50.665812    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle podName:f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e nodeName:}" failed. No retries permitted until 2026-07-16 11:51:58.665794689 +0000 UTC m=+59.966571765 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle") pod "router-default-85b9cb9c46-gmf4x" (UID: "f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e") : configmap references non-existent config key: service-ca.crt
Jul 16 11:51:50.665898 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:50.665841    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 11:51:50.665898 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:50.665877    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs podName:f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e nodeName:}" failed. No retries permitted until 2026-07-16 11:51:58.665866769 +0000 UTC m=+59.966643844 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs") pod "router-default-85b9cb9c46-gmf4x" (UID: "f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e") : secret "router-metrics-certs-default" not found
Jul 16 11:51:53.485893 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:53.485856    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9kmwb\" (UID: \"4391689c-b45b-42f6-9e95-b7ea48a4e131\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb"
Jul 16 11:51:53.486375 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:53.485989    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 11:51:53.486375 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:53.486065    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls podName:4391689c-b45b-42f6-9e95-b7ea48a4e131 nodeName:}" failed. No retries permitted until 2026-07-16 11:52:01.486029406 +0000 UTC m=+62.786806483 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls") pod "cluster-samples-operator-65db858894-9kmwb" (UID: "4391689c-b45b-42f6-9e95-b7ea48a4e131") : secret "samples-operator-tls" not found
Jul 16 11:51:54.494501 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:54.494452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:54.494852 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:54.494519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:51:54.494852 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:54.494577    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 11:51:54.494852 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:54.494617    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 11:51:54.494852 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:54.494646    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-rbac-proxy-cm podName:81f767f1-2e6f-4ad1-b92d-612c527a97b3 nodeName:}" failed. No retries permitted until 2026-07-16 11:52:02.494629801 +0000 UTC m=+63.795406878 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rr62s" (UID: "81f767f1-2e6f-4ad1-b92d-612c527a97b3") : configmap "kube-rbac-proxy" not found
Jul 16 11:51:54.494852 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:54.494679    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls podName:81f767f1-2e6f-4ad1-b92d-612c527a97b3 nodeName:}" failed. No retries permitted until 2026-07-16 11:52:02.494664962 +0000 UTC m=+63.795442038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rr62s" (UID: "81f767f1-2e6f-4ad1-b92d-612c527a97b3") : secret "insights-runtime-extractor-tls" not found
Jul 16 11:51:58.624964 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:58.624931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-vpthq\" (UID: \"715b8182-f04a-4e70-9766-1f4075a48dab\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq"
Jul 16 11:51:58.625400 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:58.625089    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 11:51:58.625400 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:58.625146    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls podName:715b8182-f04a-4e70-9766-1f4075a48dab nodeName:}" failed. No retries permitted until 2026-07-16 11:52:14.625131888 +0000 UTC m=+75.925908964 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-vpthq" (UID: "715b8182-f04a-4e70-9766-1f4075a48dab") : secret "cluster-monitoring-operator-tls" not found
Jul 16 11:51:58.726309 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:58.726279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:58.726484 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:58.726421    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle podName:f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e nodeName:}" failed. No retries permitted until 2026-07-16 11:52:14.726404677 +0000 UTC m=+76.027181752 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle") pod "router-default-85b9cb9c46-gmf4x" (UID: "f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e") : configmap references non-existent config key: service-ca.crt
Jul 16 11:51:58.726484 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:51:58.726463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:51:58.726576 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:58.726542    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 11:51:58.726576 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:51:58.726569    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs podName:f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e nodeName:}" failed. No retries permitted until 2026-07-16 11:52:14.726561432 +0000 UTC m=+76.027338508 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs") pod "router-default-85b9cb9c46-gmf4x" (UID: "f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e") : secret "router-metrics-certs-default" not found
Jul 16 11:52:00.497645 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:00.497603    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kbw5x"
Jul 16 11:52:01.548547 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:01.548507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9kmwb\" (UID: \"4391689c-b45b-42f6-9e95-b7ea48a4e131\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb"
Jul 16 11:52:01.548920 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:52:01.548658    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 11:52:01.548920 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:52:01.548729    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls podName:4391689c-b45b-42f6-9e95-b7ea48a4e131 nodeName:}" failed. No retries permitted until 2026-07-16 11:52:17.548709958 +0000 UTC m=+78.849487033 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls") pod "cluster-samples-operator-65db858894-9kmwb" (UID: "4391689c-b45b-42f6-9e95-b7ea48a4e131") : secret "samples-operator-tls" not found
Jul 16 11:52:02.556714 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:02.556660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:52:02.556714 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:02.556730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:52:02.557306 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:52:02.556827    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 11:52:02.557306 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:52:02.556880    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls podName:81f767f1-2e6f-4ad1-b92d-612c527a97b3 nodeName:}" failed. No retries permitted until 2026-07-16 11:52:18.556865445 +0000 UTC m=+79.857642521 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rr62s" (UID: "81f767f1-2e6f-4ad1-b92d-612c527a97b3") : secret "insights-runtime-extractor-tls" not found
Jul 16 11:52:02.557410 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:02.557340    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81f767f1-2e6f-4ad1-b92d-612c527a97b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:52:03.867294 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:03.867255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret\") pod \"global-pull-secret-syncer-w5wfm\" (UID: \"32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7\") " pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:52:03.867684 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:03.867321    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs\") pod \"network-metrics-daemon-4p7gl\" (UID: \"18353093-2e17-408e-b541-5dfb339343f0\") " pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:52:03.869911 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:03.869892    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 11:52:03.870086 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:03.870069    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 11:52:03.878062 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:52:03.878029    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 16 11:52:03.878151 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:52:03.878136    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs podName:18353093-2e17-408e-b541-5dfb339343f0 nodeName:}" failed. No retries permitted until 2026-07-16 11:53:07.878113778 +0000 UTC m=+129.178890855 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs") pod "network-metrics-daemon-4p7gl" (UID: "18353093-2e17-408e-b541-5dfb339343f0") : secret "metrics-daemon-secret" not found
Jul 16 11:52:03.880203 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:03.880171    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7-original-pull-secret\") pod \"global-pull-secret-syncer-w5wfm\" (UID: \"32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7\") " pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:52:03.912855 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:03.912830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5wfm"
Jul 16 11:52:03.968307 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:03.968261    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4gmx\" (UniqueName: \"kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx\") pod \"network-check-target-8p6dq\" (UID: \"3768287e-bbac-495d-a015-f6fca7d1ec36\") " pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:52:03.971610 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:03.971520    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 11:52:03.981273 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:03.981249    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 11:52:03.991672 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:03.991648    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4gmx\" (UniqueName: \"kubernetes.io/projected/3768287e-bbac-495d-a015-f6fca7d1ec36-kube-api-access-x4gmx\") pod \"network-check-target-8p6dq\" (UID: \"3768287e-bbac-495d-a015-f6fca7d1ec36\") " pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:52:04.034450 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:04.034418    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w5wfm"]
Jul 16 11:52:04.038837 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:52:04.038812    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32fffc0e_1b8f_44f1_ab44_b4b0ac9820d7.slice/crio-d723b51b608d644eb453bd4546d61304a0d224b235c6c07cb4ffe5296e44777f WatchSource:0}: Error finding container d723b51b608d644eb453bd4546d61304a0d224b235c6c07cb4ffe5296e44777f: Status 404 returned error can't find the container with id d723b51b608d644eb453bd4546d61304a0d224b235c6c07cb4ffe5296e44777f
Jul 16 11:52:04.209410 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:04.209330    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gtptt\""
Jul 16 11:52:04.217502 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:04.217483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:52:04.345242 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:04.345214    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8p6dq"]
Jul 16 11:52:04.349193 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:52:04.349169    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3768287e_bbac_495d_a015_f6fca7d1ec36.slice/crio-f4789cd90f95ab1b232fb3704e764d691d73775877c4fa8b3574c55357759d16 WatchSource:0}: Error finding container f4789cd90f95ab1b232fb3704e764d691d73775877c4fa8b3574c55357759d16: Status 404 returned error can't find the container with id f4789cd90f95ab1b232fb3704e764d691d73775877c4fa8b3574c55357759d16
Jul 16 11:52:04.581189 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:04.581154    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w5wfm" event={"ID":"32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7","Type":"ContainerStarted","Data":"d723b51b608d644eb453bd4546d61304a0d224b235c6c07cb4ffe5296e44777f"}
Jul 16 11:52:04.582509 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:04.582484    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8p6dq" event={"ID":"3768287e-bbac-495d-a015-f6fca7d1ec36","Type":"ContainerStarted","Data":"f4789cd90f95ab1b232fb3704e764d691d73775877c4fa8b3574c55357759d16"}
Jul 16 11:52:06.390377 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:06.390335    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls\") pod \"dns-default-qm4wx\" (UID: \"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a\") " pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:52:06.390803 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:52:06.390503    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 11:52:06.390803 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:52:06.390583    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls podName:2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a nodeName:}" failed. No retries permitted until 2026-07-16 11:52:38.390563293 +0000 UTC m=+99.691340374 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls") pod "dns-default-qm4wx" (UID: "2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a") : secret "dns-default-metrics-tls" not found
Jul 16 11:52:09.598541 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:09.598503    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8p6dq" event={"ID":"3768287e-bbac-495d-a015-f6fca7d1ec36","Type":"ContainerStarted","Data":"97cd9e63e23ecc5acfda3ea9da76cedc7048a5159a1ec9963bc9df5b0e0ac6fe"}
Jul 16 11:52:09.599010 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:09.598584    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:52:09.600019 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:09.599996    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w5wfm" event={"ID":"32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7","Type":"ContainerStarted","Data":"fe0a6c571280956c4b11951280b4ecfca40c9740510f472fe6a28ca2ba352e76"}
Jul 16 11:52:09.617568 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:09.617523    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8p6dq" podStartSLOduration=64.904839968 podStartE2EDuration="1m9.617510636s" totalImagesPullingTime="4.712670668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:52:09.617287276 +0000 UTC m=+70.918064394" watchObservedRunningTime="2026-07-16 11:52:09.617510636 +0000 UTC m=+70.918287746"
Jul 16 11:52:09.638500 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:09.638458    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-w5wfm" podStartSLOduration=64.602039903 podStartE2EDuration="1m9.638446984s" totalImagesPullingTime="5.036407081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:52:09.637185774 +0000 UTC m=+70.937962873" watchObservedRunningTime="2026-07-16 11:52:09.638446984 +0000 UTC m=+70.939224127"
Jul 16 11:52:14.661805 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:14.661760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-vpthq\" (UID: \"715b8182-f04a-4e70-9766-1f4075a48dab\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq"
Jul 16 11:52:14.664111 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:14.664078    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/715b8182-f04a-4e70-9766-1f4075a48dab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-vpthq\" (UID: \"715b8182-f04a-4e70-9766-1f4075a48dab\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq"
Jul 16 11:52:14.762548 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:14.762523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:52:14.762676 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:14.762582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:52:14.763114 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:14.763096    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-service-ca-bundle\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:52:14.764738 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:14.764718    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e-metrics-certs\") pod \"router-default-85b9cb9c46-gmf4x\" (UID: \"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e\") " pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:52:14.906408 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:14.906383    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-65t59\""
Jul 16 11:52:14.914795 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:14.914756    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq"
Jul 16 11:52:14.946654 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:14.946621    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vct7t\""
Jul 16 11:52:14.953890 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:14.953868    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:52:15.052878 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:15.052851    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq"]
Jul 16 11:52:15.058117 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:52:15.057792    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715b8182_f04a_4e70_9766_1f4075a48dab.slice/crio-60a01f096c57b536faaf8108ba01bbf7be1442c47f030d1ea4e1c10997c50175 WatchSource:0}: Error finding container 60a01f096c57b536faaf8108ba01bbf7be1442c47f030d1ea4e1c10997c50175: Status 404 returned error can't find the container with id 60a01f096c57b536faaf8108ba01bbf7be1442c47f030d1ea4e1c10997c50175
Jul 16 11:52:15.107923 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:15.107901    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-85b9cb9c46-gmf4x"]
Jul 16 11:52:15.110806 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:52:15.110787    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01a2273_37fb_49cb_a9a3_17ebbd6f8f3e.slice/crio-f09b81d4e8c10da324b4ce74c827b717731f19ab57768266f88fb2a198889bcf WatchSource:0}: Error finding container f09b81d4e8c10da324b4ce74c827b717731f19ab57768266f88fb2a198889bcf: Status 404 returned error can't find the container with id f09b81d4e8c10da324b4ce74c827b717731f19ab57768266f88fb2a198889bcf
Jul 16 11:52:15.617002 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:15.616962    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-85b9cb9c46-gmf4x" event={"ID":"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e","Type":"ContainerStarted","Data":"dbdc0383c28df01f8d3e72c17b451413492fe7e5f2425b6eb49bcc70b313f104"}
Jul 16 11:52:15.617203 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:15.617028    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-85b9cb9c46-gmf4x" event={"ID":"f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e","Type":"ContainerStarted","Data":"f09b81d4e8c10da324b4ce74c827b717731f19ab57768266f88fb2a198889bcf"}
Jul 16 11:52:15.618196 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:15.618171    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq" event={"ID":"715b8182-f04a-4e70-9766-1f4075a48dab","Type":"ContainerStarted","Data":"60a01f096c57b536faaf8108ba01bbf7be1442c47f030d1ea4e1c10997c50175"}
Jul 16 11:52:15.659495 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:15.659439    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-85b9cb9c46-gmf4x" podStartSLOduration=33.659418674 podStartE2EDuration="33.659418674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:52:15.6575312 +0000 UTC m=+76.958308299" watchObservedRunningTime="2026-07-16 11:52:15.659418674 +0000 UTC m=+76.960195773"
Jul 16 11:52:15.954796 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:15.954719    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:52:15.957789 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:15.957761    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:52:16.622466 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:16.622431    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:52:16.623822 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:16.623798    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-85b9cb9c46-gmf4x"
Jul 16 11:52:17.586549 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:17.586511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9kmwb\" (UID: \"4391689c-b45b-42f6-9e95-b7ea48a4e131\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb"
Jul 16 11:52:17.588982 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:17.588946    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4391689c-b45b-42f6-9e95-b7ea48a4e131-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9kmwb\" (UID: \"4391689c-b45b-42f6-9e95-b7ea48a4e131\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb"
Jul 16 11:52:17.626805 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:17.626766    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq" event={"ID":"715b8182-f04a-4e70-9766-1f4075a48dab","Type":"ContainerStarted","Data":"bf6f556fa84e95452f64eda2d6b6a9f326b9c5dc5d950e9d9c94e68dc3d959b4"}
Jul 16 11:52:17.815445 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:17.815418    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cvcg7\""
Jul 16 11:52:17.821818 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:17.821796    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb"
Jul 16 11:52:17.956198 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:17.956143    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vpthq" podStartSLOduration=34.07365083 podStartE2EDuration="35.956125442s" totalImagesPullingTime="1.882474612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:52:17.671868056 +0000 UTC m=+78.972645155" watchObservedRunningTime="2026-07-16 11:52:17.956125442 +0000 UTC m=+79.256902543"
Jul 16 11:52:17.956518 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:17.956502    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb"]
Jul 16 11:52:18.594173 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:18.594141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:52:18.596273 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:18.596248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81f767f1-2e6f-4ad1-b92d-612c527a97b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rr62s\" (UID: \"81f767f1-2e6f-4ad1-b92d-612c527a97b3\") " pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:52:18.631091 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:18.631063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb" event={"ID":"4391689c-b45b-42f6-9e95-b7ea48a4e131","Type":"ContainerStarted","Data":"9d855f2ade0811b8cd16e54361904c7475bfced25caee842521465e0e09ceb28"}
Jul 16 11:52:18.808920 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:18.808894    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qlctg\""
Jul 16 11:52:18.816219 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:18.816200    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rr62s"
Jul 16 11:52:18.952238 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:18.952159    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rr62s"]
Jul 16 11:52:18.954644 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:52:18.954619    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f767f1_2e6f_4ad1_b92d_612c527a97b3.slice/crio-937897bb96780bb2022660d2552c9a2f00ec84ba579844446fa15b0e58b9ec36 WatchSource:0}: Error finding container 937897bb96780bb2022660d2552c9a2f00ec84ba579844446fa15b0e58b9ec36: Status 404 returned error can't find the container with id 937897bb96780bb2022660d2552c9a2f00ec84ba579844446fa15b0e58b9ec36
Jul 16 11:52:19.638908 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:19.638869    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rr62s" event={"ID":"81f767f1-2e6f-4ad1-b92d-612c527a97b3","Type":"ContainerStarted","Data":"937897bb96780bb2022660d2552c9a2f00ec84ba579844446fa15b0e58b9ec36"}
Jul 16 11:52:24.657978 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:24.657939    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rr62s" event={"ID":"81f767f1-2e6f-4ad1-b92d-612c527a97b3","Type":"ContainerStarted","Data":"33253b4f0c233b2bcd23a8ae7ccbe0b8d0a15a370f4aad6e95275d922ab8962d"}
Jul 16 11:52:26.664996 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:26.664956    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb" event={"ID":"4391689c-b45b-42f6-9e95-b7ea48a4e131","Type":"ContainerStarted","Data":"8b4faea475f02ff7770e2e7a0d8dfd1838e522f07f1f6675d515eaa17c88922b"}
Jul 16 11:52:26.665403 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:26.665063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb" event={"ID":"4391689c-b45b-42f6-9e95-b7ea48a4e131","Type":"ContainerStarted","Data":"a0a44432dddf73cb086b506219dd4e1d569c78de5959534523da99bcc8e2fac9"}
Jul 16 11:52:26.685441 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:26.685384    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9kmwb" podStartSLOduration=34.059987849 podStartE2EDuration="41.685363272s" totalImagesPullingTime="7.625375423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:52:26.684756788 +0000 UTC m=+87.985533886" watchObservedRunningTime="2026-07-16 11:52:26.685363272 +0000 UTC m=+87.986140371"
Jul 16 11:52:28.979400 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:28.979369    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"]
Jul 16 11:52:28.989971 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:28.989850    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"
Jul 16 11:52:28.992849 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:28.992826    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 16 11:52:28.993674 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:28.993624    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 16 11:52:28.994332 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:28.994309    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 16 11:52:29.000100 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.000079    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 16 11:52:29.000606 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.000583    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"]
Jul 16 11:52:29.000699 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.000615    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 16 11:52:29.000699 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.000682    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 16 11:52:29.000805 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.000761    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 16 11:52:29.080244 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.080211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/434e4ab5-b283-4750-a9a8-ea1a93f1cb1f-hub\") pod \"cluster-proxy-proxy-agent-748fdfcddb-rgqxw\" (UID: \"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"
Jul 16 11:52:29.080404 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.080262    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/434e4ab5-b283-4750-a9a8-ea1a93f1cb1f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-748fdfcddb-rgqxw\" (UID: \"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"
Jul 16 11:52:29.080404 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.080368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/434e4ab5-b283-4750-a9a8-ea1a93f1cb1f-ca\") pod \"cluster-proxy-proxy-agent-748fdfcddb-rgqxw\" (UID: \"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"
Jul 16 11:52:29.080510 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.080422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/434e4ab5-b283-4750-a9a8-ea1a93f1cb1f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-748fdfcddb-rgqxw\" (UID: \"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"
Jul 16 11:52:29.080510 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.080480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/434e4ab5-b283-4750-a9a8-ea1a93f1cb1f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-748fdfcddb-rgqxw\" (UID: \"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"
Jul 16 11:52:29.080593 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.080536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtgfn\" (UniqueName: \"kubernetes.io/projected/434e4ab5-b283-4750-a9a8-ea1a93f1cb1f-kube-api-access-gtgfn\") pod \"cluster-proxy-proxy-agent-748fdfcddb-rgqxw\" (UID: \"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"
Jul 16 11:52:29.181135 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.181087    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/434e4ab5-b283-4750-a9a8-ea1a93f1cb1f-ca\") pod \"cluster-proxy-proxy-agent-748fdfcddb-rgqxw\" (UID: \"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"
Jul 16 11:52:29.181321 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.181144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/434e4ab5-b283-4750-a9a8-ea1a93f1cb1f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-748fdfcddb-rgqxw\" (UID: \"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"
Jul 16 11:52:29.181321 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.181194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/434e4ab5-b283-4750-a9a8-ea1a93f1cb1f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-748fdfcddb-rgqxw\" (UID: \"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"
Jul 16 11:52:29.181321 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.181242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtgfn\" (UniqueName: \"kubernetes.io/projected/434e4ab5-b283-4750-a9a8-ea1a93f1cb1f-kube-api-access-gtgfn\") pod \"cluster-proxy-proxy-agent-748fdfcddb-rgqxw\" (UID: \"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"
Jul 16 11:52:29.181321 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.181289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/434e4ab5-b283-4750-a9a8-ea1a93f1cb1f-hub\") pod \"cluster-proxy-proxy-agent-748fdfcddb-rgqxw\" (UID: \"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"
Jul 16 11:52:29.181527 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.181332    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/434e4ab5-b283-4750-a9a8-ea1a93f1cb1f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-748fdfcddb-rgqxw\" (UID: \"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"
Jul 16 11:52:29.182363 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.182318    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/434e4ab5-b283-4750-a9a8-ea1a93f1cb1f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-748fdfcddb-rgqxw\" (UID: \"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"
Jul 16 11:52:29.184359 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.184311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/434e4ab5-b283-4750-a9a8-ea1a93f1cb1f-ca\") pod \"cluster-proxy-proxy-agent-748fdfcddb-rgqxw\" (UID: \"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"
Jul 16 11:52:29.184862 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.184834    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/434e4ab5-b283-4750-a9a8-ea1a93f1cb1f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-748fdfcddb-rgqxw\" (UID: \"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"
Jul 16 11:52:29.185132 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.185107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/434e4ab5-b283-4750-a9a8-ea1a93f1cb1f-hub\") pod \"cluster-proxy-proxy-agent-748fdfcddb-rgqxw\" (UID: \"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"
Jul 16 11:52:29.185766 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.185743    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/434e4ab5-b283-4750-a9a8-ea1a93f1cb1f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-748fdfcddb-rgqxw\" (UID: \"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"
Jul 16 11:52:29.214239 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.214210    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtgfn\" (UniqueName: \"kubernetes.io/projected/434e4ab5-b283-4750-a9a8-ea1a93f1cb1f-kube-api-access-gtgfn\") pod \"cluster-proxy-proxy-agent-748fdfcddb-rgqxw\" (UID: \"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"
Jul 16 11:52:29.306181 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:29.306159    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"
Jul 16 11:52:30.518237 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:30.518208    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw"]
Jul 16 11:52:30.521529 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:52:30.521503    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod434e4ab5_b283_4750_a9a8_ea1a93f1cb1f.slice/crio-f4722388dd0ff62decd1e9f87380b1a441692a8ba856866888e02c965b5c73cd WatchSource:0}: Error finding container f4722388dd0ff62decd1e9f87380b1a441692a8ba856866888e02c965b5c73cd: Status 404 returned error can't find the container with id f4722388dd0ff62decd1e9f87380b1a441692a8ba856866888e02c965b5c73cd
Jul 16 11:52:30.679905 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:30.679843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rr62s" event={"ID":"81f767f1-2e6f-4ad1-b92d-612c527a97b3","Type":"ContainerStarted","Data":"c609e18d0e1a3394f5a6181756a4aa619a584cec517681ffa85fe0c86c5ddaea"}
Jul 16 11:52:30.680888 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:30.680865    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw" event={"ID":"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f","Type":"ContainerStarted","Data":"f4722388dd0ff62decd1e9f87380b1a441692a8ba856866888e02c965b5c73cd"}
Jul 16 11:52:36.709799 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:36.709762    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rr62s" event={"ID":"81f767f1-2e6f-4ad1-b92d-612c527a97b3","Type":"ContainerStarted","Data":"1f9f11fd5163f4042de841aaa4d8be63af9624b1c2c0f7e42d0cda1257a7eac8"}
Jul 16 11:52:36.711833 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:36.711811    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw" event={"ID":"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f","Type":"ContainerStarted","Data":"a5e07a18c437f243d72c39ceae4e53ca445cda3babcbdb0391f2e95c6728c55b"}
Jul 16 11:52:36.711973 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:36.711843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw" event={"ID":"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f","Type":"ContainerStarted","Data":"f542ec71ab13db9db5fe2eb900833e434aa9a71b050e9b02d8d8541988187355"}
Jul 16 11:52:36.711973 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:36.711855    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw" event={"ID":"434e4ab5-b283-4750-a9a8-ea1a93f1cb1f","Type":"ContainerStarted","Data":"71732db1aef0495788b378ad3dca1e1c95176574928b8f5c394c78c250ff5c64"}
Jul 16 11:52:36.732703 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:36.732639    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rr62s" podStartSLOduration=33.995169608 podStartE2EDuration="50.732624902s" totalImagesPullingTime="16.737455294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:51:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:52:36.731253724 +0000 UTC m=+98.032030821" watchObservedRunningTime="2026-07-16 11:52:36.732624902 +0000 UTC m=+98.033402002"
Jul 16 11:52:36.757706 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:36.757652    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748fdfcddb-rgqxw" podStartSLOduration=3.466101833 podStartE2EDuration="8.757631291s" totalImagesPullingTime="5.291529458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:52:36.756815771 +0000 UTC m=+98.057592868" watchObservedRunningTime="2026-07-16 11:52:36.757631291 +0000 UTC m=+98.058408391"
Jul 16 11:52:38.458160 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:38.458127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls\") pod \"dns-default-qm4wx\" (UID: \"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a\") " pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:52:38.460439 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:38.460407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a-metrics-tls\") pod \"dns-default-qm4wx\" (UID: \"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a\") " pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:52:38.462831 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:38.462813    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b759k\""
Jul 16 11:52:38.470875 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:38.470851    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:52:38.623790 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:38.623768    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qm4wx"]
Jul 16 11:52:38.626191 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:52:38.626165    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8ed3c2_9f66_4cb3_be0d_a58c0f1f1a5a.slice/crio-88dcb69959fe0e08b98e737335d1195eb6731352e6a5fa294cf46f582a96c67f WatchSource:0}: Error finding container 88dcb69959fe0e08b98e737335d1195eb6731352e6a5fa294cf46f582a96c67f: Status 404 returned error can't find the container with id 88dcb69959fe0e08b98e737335d1195eb6731352e6a5fa294cf46f582a96c67f
Jul 16 11:52:38.718886 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:38.718830    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qm4wx" event={"ID":"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a","Type":"ContainerStarted","Data":"88dcb69959fe0e08b98e737335d1195eb6731352e6a5fa294cf46f582a96c67f"}
Jul 16 11:52:40.605304 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:40.605274    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8p6dq"
Jul 16 11:52:41.463216 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.463176    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zkrrr"]
Jul 16 11:52:41.465821 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.465806    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.471129 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.471101    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 11:52:41.471129 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.471119    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 11:52:41.471438 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.471418    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 11:52:41.471510 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.471483    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vz9xv\""
Jul 16 11:52:41.471575 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.471485    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 11:52:41.580826 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.580789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f9k6\" (UniqueName: \"kubernetes.io/projected/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-kube-api-access-9f9k6\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.580994 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.580835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.580994 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.580857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-node-exporter-wtmp\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.580994 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.580936    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-sys\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.580994 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.580973    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-node-exporter-accelerators-collector-config\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.581153 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.581022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-root\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.581153 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.581077    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-node-exporter-textfile\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.581153 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.581096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-node-exporter-tls\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.581153 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.581111    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-metrics-client-ca\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.682137 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.682097    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-node-exporter-textfile\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.682137 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.682138    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-node-exporter-tls\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.682624 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.682160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-metrics-client-ca\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.682624 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.682219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f9k6\" (UniqueName: \"kubernetes.io/projected/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-kube-api-access-9f9k6\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.682624 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.682251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.682624 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.682282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-node-exporter-wtmp\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.682624 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.682315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-sys\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.682624 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.682349    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-node-exporter-accelerators-collector-config\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.682624 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.682390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-root\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.682624 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.682418    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-sys\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.682624 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.682483    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-node-exporter-textfile\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.682624 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.682545    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-root\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.682930 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.682786    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-metrics-client-ca\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.682930 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.682862    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-node-exporter-accelerators-collector-config\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.682994 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.682931    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-node-exporter-wtmp\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.684672 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.684655    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-node-exporter-tls\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.684724 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.684690    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.695056 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.695014    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f9k6\" (UniqueName: \"kubernetes.io/projected/a3bd6541-f3e1-41ac-8693-ba50e1366d9b-kube-api-access-9f9k6\") pod \"node-exporter-zkrrr\" (UID: \"a3bd6541-f3e1-41ac-8693-ba50e1366d9b\") " pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.777146 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:41.777088    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zkrrr"
Jul 16 11:52:41.786517 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:52:41.786490    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3bd6541_f3e1_41ac_8693_ba50e1366d9b.slice/crio-41fa48a5473a67063d82b47cec3949cf8bfe4fc749a229c90713ef81ffa6bc73 WatchSource:0}: Error finding container 41fa48a5473a67063d82b47cec3949cf8bfe4fc749a229c90713ef81ffa6bc73: Status 404 returned error can't find the container with id 41fa48a5473a67063d82b47cec3949cf8bfe4fc749a229c90713ef81ffa6bc73
Jul 16 11:52:42.735927 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:42.735887    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zkrrr" event={"ID":"a3bd6541-f3e1-41ac-8693-ba50e1366d9b","Type":"ContainerStarted","Data":"41fa48a5473a67063d82b47cec3949cf8bfe4fc749a229c90713ef81ffa6bc73"}
Jul 16 11:52:43.450133 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:43.450100    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-bqhbq"]
Jul 16 11:52:43.452063 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:43.452028    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-bqhbq"
Jul 16 11:52:43.457482 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:43.457426    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 16 11:52:43.457482 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:43.457484    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-hv29m\""
Jul 16 11:52:43.457653 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:43.457494    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 16 11:52:43.478821 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:43.478796    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-bqhbq"]
Jul 16 11:52:43.498942 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:43.498918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5j5r\" (UniqueName: \"kubernetes.io/projected/940b7de4-6a55-4b35-8c54-adb00b18b803-kube-api-access-h5j5r\") pod \"downloads-68c9cbb497-bqhbq\" (UID: \"940b7de4-6a55-4b35-8c54-adb00b18b803\") " pod="openshift-console/downloads-68c9cbb497-bqhbq"
Jul 16 11:52:43.600314 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:43.600287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5j5r\" (UniqueName: \"kubernetes.io/projected/940b7de4-6a55-4b35-8c54-adb00b18b803-kube-api-access-h5j5r\") pod \"downloads-68c9cbb497-bqhbq\" (UID: \"940b7de4-6a55-4b35-8c54-adb00b18b803\") " pod="openshift-console/downloads-68c9cbb497-bqhbq"
Jul 16 11:52:43.619773 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:43.619748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5j5r\" (UniqueName: \"kubernetes.io/projected/940b7de4-6a55-4b35-8c54-adb00b18b803-kube-api-access-h5j5r\") pod \"downloads-68c9cbb497-bqhbq\" (UID: \"940b7de4-6a55-4b35-8c54-adb00b18b803\") " pod="openshift-console/downloads-68c9cbb497-bqhbq"
Jul 16 11:52:43.741068 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:43.740972    2578 generic.go:356] "Generic (PLEG): container finished" podID="a3bd6541-f3e1-41ac-8693-ba50e1366d9b" containerID="bd58f2189544937420096e8ef9463b76aa6680391d9446509297a7df265186f3" exitCode=0
Jul 16 11:52:43.741462 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:43.741079    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zkrrr" event={"ID":"a3bd6541-f3e1-41ac-8693-ba50e1366d9b","Type":"ContainerDied","Data":"bd58f2189544937420096e8ef9463b76aa6680391d9446509297a7df265186f3"}
Jul 16 11:52:43.742896 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:43.742872    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qm4wx" event={"ID":"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a","Type":"ContainerStarted","Data":"fe7b2812d7f0645b8f747a9729e442a9361a85b3cd3bfb242ebd281bb4521de2"}
Jul 16 11:52:43.742988 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:43.742913    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qm4wx" event={"ID":"2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a","Type":"ContainerStarted","Data":"3673206e72bab078bd62dbfa11ef063e86235669448a9f0879c4c42f19dc6168"}
Jul 16 11:52:43.743031 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:43.742979    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:52:43.760639 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:43.760620    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-bqhbq"
Jul 16 11:52:43.805594 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:43.805531    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qm4wx" podStartSLOduration=65.219874294 podStartE2EDuration="1m9.805510414s" totalImagesPullingTime="4.58563612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:52:43.803493165 +0000 UTC m=+105.104270266" watchObservedRunningTime="2026-07-16 11:52:43.805510414 +0000 UTC m=+105.106287514"
Jul 16 11:52:43.921713 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:43.921686    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-bqhbq"]
Jul 16 11:52:43.924984 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:52:43.924936    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940b7de4_6a55_4b35_8c54_adb00b18b803.slice/crio-d95ee7f449136fb47f7d1b80c1e108b71139e5ff61a3df0efec2a60671025793 WatchSource:0}: Error finding container d95ee7f449136fb47f7d1b80c1e108b71139e5ff61a3df0efec2a60671025793: Status 404 returned error can't find the container with id d95ee7f449136fb47f7d1b80c1e108b71139e5ff61a3df0efec2a60671025793
Jul 16 11:52:44.747858 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:44.747820    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-bqhbq" event={"ID":"940b7de4-6a55-4b35-8c54-adb00b18b803","Type":"ContainerStarted","Data":"d95ee7f449136fb47f7d1b80c1e108b71139e5ff61a3df0efec2a60671025793"}
Jul 16 11:52:44.749830 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:44.749800    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zkrrr" event={"ID":"a3bd6541-f3e1-41ac-8693-ba50e1366d9b","Type":"ContainerStarted","Data":"2857cbdc3b3983267d1f2448425e02e0e129a404c73f935f2f00dba50a940921"}
Jul 16 11:52:44.749955 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:44.749850    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zkrrr" event={"ID":"a3bd6541-f3e1-41ac-8693-ba50e1366d9b","Type":"ContainerStarted","Data":"ae8ddf0c7afa5bb34f27f304206d69706f57c30e0796a6b1e59c95f5c92fa4e6"}
Jul 16 11:52:53.018403 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.018339    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zkrrr" podStartSLOduration=10.521604599 podStartE2EDuration="12.018317496s" totalImagesPullingTime="1.418470276s" totalInitContainerRuntime="78.242621ms" isStatefulPod=true podCreationTimestamp="2026-07-16 11:52:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:52:44.776300265 +0000 UTC m=+106.077077363" watchObservedRunningTime="2026-07-16 11:52:53.018317496 +0000 UTC m=+114.319094597"
Jul 16 11:52:53.019615 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.019597    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b699464c7-xztxx"]
Jul 16 11:52:53.025899 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.025877    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:52:53.029336 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.029317    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 16 11:52:53.029457 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.029321    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rcddz\""
Jul 16 11:52:53.030329 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.030286    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 16 11:52:53.030452 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.030304    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 16 11:52:53.037364 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.037344    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 16 11:52:53.038614 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.038597    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 16 11:52:53.070282 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.070255    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7b0103f-06a5-4424-81ff-66cb96e40efe-console-serving-cert\") pod \"console-5b699464c7-xztxx\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") " pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:52:53.070400 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.070292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e7b0103f-06a5-4424-81ff-66cb96e40efe-console-oauth-config\") pod \"console-5b699464c7-xztxx\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") " pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:52:53.070452 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.070400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e7b0103f-06a5-4424-81ff-66cb96e40efe-oauth-serving-cert\") pod \"console-5b699464c7-xztxx\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") " pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:52:53.070498 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.070449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzwzp\" (UniqueName: \"kubernetes.io/projected/e7b0103f-06a5-4424-81ff-66cb96e40efe-kube-api-access-rzwzp\") pod \"console-5b699464c7-xztxx\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") " pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:52:53.070498 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.070476    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e7b0103f-06a5-4424-81ff-66cb96e40efe-console-config\") pod \"console-5b699464c7-xztxx\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") " pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:52:53.070577 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.070525    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b0103f-06a5-4424-81ff-66cb96e40efe-service-ca\") pod \"console-5b699464c7-xztxx\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") " pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:52:53.074968 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.074946    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b699464c7-xztxx"]
Jul 16 11:52:53.170834 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.170808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b0103f-06a5-4424-81ff-66cb96e40efe-service-ca\") pod \"console-5b699464c7-xztxx\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") " pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:52:53.170985 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.170853    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7b0103f-06a5-4424-81ff-66cb96e40efe-console-serving-cert\") pod \"console-5b699464c7-xztxx\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") " pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:52:53.171065 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.170988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e7b0103f-06a5-4424-81ff-66cb96e40efe-console-oauth-config\") pod \"console-5b699464c7-xztxx\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") " pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:52:53.171125 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.171099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e7b0103f-06a5-4424-81ff-66cb96e40efe-oauth-serving-cert\") pod \"console-5b699464c7-xztxx\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") " pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:52:53.171177 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.171140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzwzp\" (UniqueName: \"kubernetes.io/projected/e7b0103f-06a5-4424-81ff-66cb96e40efe-kube-api-access-rzwzp\") pod \"console-5b699464c7-xztxx\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") " pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:52:53.171177 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.171167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e7b0103f-06a5-4424-81ff-66cb96e40efe-console-config\") pod \"console-5b699464c7-xztxx\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") " pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:52:53.171681 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.171654    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b0103f-06a5-4424-81ff-66cb96e40efe-service-ca\") pod \"console-5b699464c7-xztxx\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") " pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:52:53.171813 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.171773    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e7b0103f-06a5-4424-81ff-66cb96e40efe-oauth-serving-cert\") pod \"console-5b699464c7-xztxx\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") " pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:52:53.171911 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.171885    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e7b0103f-06a5-4424-81ff-66cb96e40efe-console-config\") pod \"console-5b699464c7-xztxx\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") " pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:52:53.173724 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.173693    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e7b0103f-06a5-4424-81ff-66cb96e40efe-console-oauth-config\") pod \"console-5b699464c7-xztxx\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") " pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:52:53.173845 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.173824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7b0103f-06a5-4424-81ff-66cb96e40efe-console-serving-cert\") pod \"console-5b699464c7-xztxx\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") " pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:52:53.197320 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.197297    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzwzp\" (UniqueName: \"kubernetes.io/projected/e7b0103f-06a5-4424-81ff-66cb96e40efe-kube-api-access-rzwzp\") pod \"console-5b699464c7-xztxx\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") " pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:52:53.337292 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.337257    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:52:53.530207 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.530175    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b699464c7-xztxx"]
Jul 16 11:52:53.533110 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:52:53.533079    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b0103f_06a5_4424_81ff_66cb96e40efe.slice/crio-bbd2f2ae3f504f0d21124021a1e836e859b29308c646613dd52ad1d7493dfe86 WatchSource:0}: Error finding container bbd2f2ae3f504f0d21124021a1e836e859b29308c646613dd52ad1d7493dfe86: Status 404 returned error can't find the container with id bbd2f2ae3f504f0d21124021a1e836e859b29308c646613dd52ad1d7493dfe86
Jul 16 11:52:53.753074 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.752978    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qm4wx"
Jul 16 11:52:53.781084 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:52:53.781059    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b699464c7-xztxx" event={"ID":"e7b0103f-06a5-4424-81ff-66cb96e40efe","Type":"ContainerStarted","Data":"bbd2f2ae3f504f0d21124021a1e836e859b29308c646613dd52ad1d7493dfe86"}
Jul 16 11:53:04.828645 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:04.828611    2578 generic.go:356] "Generic (PLEG): container finished" podID="64f7c050-5fde-47f1-9e42-5deaca93acd7" containerID="1086c4da2c07cead574fdc97ec94c5c6cf3b5029953768fe356c3fc66cb55cf0" exitCode=0
Jul 16 11:53:04.829163 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:04.828696    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-qmc55" event={"ID":"64f7c050-5fde-47f1-9e42-5deaca93acd7","Type":"ContainerDied","Data":"1086c4da2c07cead574fdc97ec94c5c6cf3b5029953768fe356c3fc66cb55cf0"}
Jul 16 11:53:04.829163 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:04.829150    2578 scope.go:117] "RemoveContainer" containerID="1086c4da2c07cead574fdc97ec94c5c6cf3b5029953768fe356c3fc66cb55cf0"
Jul 16 11:53:04.830600 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:04.830574    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-bqhbq" event={"ID":"940b7de4-6a55-4b35-8c54-adb00b18b803","Type":"ContainerStarted","Data":"77ed559967a215d3fe2cfb609e438969fa6f1b5cd0b75a72dff411ad86414a44"}
Jul 16 11:53:04.830728 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:04.830627    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-bqhbq"
Jul 16 11:53:04.833234 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:04.833172    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b699464c7-xztxx" event={"ID":"e7b0103f-06a5-4424-81ff-66cb96e40efe","Type":"ContainerStarted","Data":"1be4d7ce5b50bf7ede15ab54350199cc4ade99a8e0841d4454b851f954cf2332"}
Jul 16 11:53:04.843276 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:04.843251    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-bqhbq"
Jul 16 11:53:04.919346 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:04.919295    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-bqhbq" podStartSLOduration=1.52891485 podStartE2EDuration="21.919276078s" totalImagesPullingTime="20.390361228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:53:04.918947706 +0000 UTC m=+126.219724807" watchObservedRunningTime="2026-07-16 11:53:04.919276078 +0000 UTC m=+126.220053178"
Jul 16 11:53:04.967949 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:04.967830    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b699464c7-xztxx" podStartSLOduration=9.678763202 podStartE2EDuration="12.967805959s" totalImagesPullingTime="3.289042757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:53:04.965688929 +0000 UTC m=+126.266466024" watchObservedRunningTime="2026-07-16 11:53:04.967805959 +0000 UTC m=+126.268583058"
Jul 16 11:53:05.839623 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:05.839578    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-qmc55" event={"ID":"64f7c050-5fde-47f1-9e42-5deaca93acd7","Type":"ContainerStarted","Data":"f0e914df5f137d561d46ae75f6c51ec51dd28626b3436b730bc7f103fb48f29c"}
Jul 16 11:53:07.038654 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:07.038617    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-vpthq_715b8182-f04a-4e70-9766-1f4075a48dab/cluster-monitoring-operator/0.log"
Jul 16 11:53:07.909015 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:07.908971    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs\") pod \"network-metrics-daemon-4p7gl\" (UID: \"18353093-2e17-408e-b541-5dfb339343f0\") " pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:53:07.911575 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:07.911549    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18353093-2e17-408e-b541-5dfb339343f0-metrics-certs\") pod \"network-metrics-daemon-4p7gl\" (UID: \"18353093-2e17-408e-b541-5dfb339343f0\") " pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:53:08.008367 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:08.008337    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vn7fb\""
Jul 16 11:53:08.015558 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:08.015527    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p7gl"
Jul 16 11:53:08.198815 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:08.198732    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4p7gl"]
Jul 16 11:53:08.202543 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:53:08.202507    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18353093_2e17_408e_b541_5dfb339343f0.slice/crio-961e5d51543519d1867abd4812118457ae3b6b4d97cf20b8172daa20d5cbb4c7 WatchSource:0}: Error finding container 961e5d51543519d1867abd4812118457ae3b6b4d97cf20b8172daa20d5cbb4c7: Status 404 returned error can't find the container with id 961e5d51543519d1867abd4812118457ae3b6b4d97cf20b8172daa20d5cbb4c7
Jul 16 11:53:08.851320 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:08.851278    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4p7gl" event={"ID":"18353093-2e17-408e-b541-5dfb339343f0","Type":"ContainerStarted","Data":"961e5d51543519d1867abd4812118457ae3b6b4d97cf20b8172daa20d5cbb4c7"}
Jul 16 11:53:09.434017 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:09.433990    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zkrrr_a3bd6541-f3e1-41ac-8693-ba50e1366d9b/init-textfile/0.log"
Jul 16 11:53:09.633260 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:09.633232    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zkrrr_a3bd6541-f3e1-41ac-8693-ba50e1366d9b/node-exporter/0.log"
Jul 16 11:53:09.833237 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:09.833203    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zkrrr_a3bd6541-f3e1-41ac-8693-ba50e1366d9b/kube-rbac-proxy/0.log"
Jul 16 11:53:11.657926 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:11.657888    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b699464c7-xztxx"]
Jul 16 11:53:12.870371 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:12.870329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4p7gl" event={"ID":"18353093-2e17-408e-b541-5dfb339343f0","Type":"ContainerStarted","Data":"58ad285f4e2832d6899d181a562482d343a1a1ecca81fa1d80867ccac0ced971"}
Jul 16 11:53:12.870844 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:12.870398    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4p7gl" event={"ID":"18353093-2e17-408e-b541-5dfb339343f0","Type":"ContainerStarted","Data":"52e8ae1e84f2fa32d0721031777e02c925ff388e228c384c57b02cb59cd9b875"}
Jul 16 11:53:13.337781 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:13.337749    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:53:14.433396 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:14.433367    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b699464c7-xztxx_e7b0103f-06a5-4424-81ff-66cb96e40efe/console/0.log"
Jul 16 11:53:14.838449 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:14.838420    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-bqhbq_940b7de4-6a55-4b35-8c54-adb00b18b803/download-server/0.log"
Jul 16 11:53:36.683096 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:36.683000    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b699464c7-xztxx" podUID="e7b0103f-06a5-4424-81ff-66cb96e40efe" containerName="console" containerID="cri-o://1be4d7ce5b50bf7ede15ab54350199cc4ade99a8e0841d4454b851f954cf2332" gracePeriod=15
Jul 16 11:53:36.952895 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:36.952825    2578 generic.go:356] "Generic (PLEG): container finished" podID="e7b0103f-06a5-4424-81ff-66cb96e40efe" containerID="1be4d7ce5b50bf7ede15ab54350199cc4ade99a8e0841d4454b851f954cf2332" exitCode=0
Jul 16 11:53:36.952895 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:36.952881    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b699464c7-xztxx" event={"ID":"e7b0103f-06a5-4424-81ff-66cb96e40efe","Type":"ContainerDied","Data":"1be4d7ce5b50bf7ede15ab54350199cc4ade99a8e0841d4454b851f954cf2332"}
Jul 16 11:53:36.953067 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:36.952926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b699464c7-xztxx" event={"ID":"e7b0103f-06a5-4424-81ff-66cb96e40efe","Type":"ContainerDied","Data":"bbd2f2ae3f504f0d21124021a1e836e859b29308c646613dd52ad1d7493dfe86"}
Jul 16 11:53:36.953067 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:36.952948    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbd2f2ae3f504f0d21124021a1e836e859b29308c646613dd52ad1d7493dfe86"
Jul 16 11:53:36.958162 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:36.958135    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:53:36.982883 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:36.982831    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4p7gl" podStartSLOduration=153.255806196 podStartE2EDuration="2m36.982817191s" totalImagesPullingTime="3.727010995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:53:12.904100652 +0000 UTC m=+134.204877753" watchObservedRunningTime="2026-07-16 11:53:36.982817191 +0000 UTC m=+158.283594289"
Jul 16 11:53:37.127483 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.127461    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b0103f-06a5-4424-81ff-66cb96e40efe-service-ca\") pod \"e7b0103f-06a5-4424-81ff-66cb96e40efe\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") "
Jul 16 11:53:37.127622 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.127522    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzwzp\" (UniqueName: \"kubernetes.io/projected/e7b0103f-06a5-4424-81ff-66cb96e40efe-kube-api-access-rzwzp\") pod \"e7b0103f-06a5-4424-81ff-66cb96e40efe\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") "
Jul 16 11:53:37.127622 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.127547    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7b0103f-06a5-4424-81ff-66cb96e40efe-console-serving-cert\") pod \"e7b0103f-06a5-4424-81ff-66cb96e40efe\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") "
Jul 16 11:53:37.127622 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.127565    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e7b0103f-06a5-4424-81ff-66cb96e40efe-oauth-serving-cert\") pod \"e7b0103f-06a5-4424-81ff-66cb96e40efe\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") "
Jul 16 11:53:37.127622 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.127587    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e7b0103f-06a5-4424-81ff-66cb96e40efe-console-config\") pod \"e7b0103f-06a5-4424-81ff-66cb96e40efe\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") "
Jul 16 11:53:37.127622 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.127610    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e7b0103f-06a5-4424-81ff-66cb96e40efe-console-oauth-config\") pod \"e7b0103f-06a5-4424-81ff-66cb96e40efe\" (UID: \"e7b0103f-06a5-4424-81ff-66cb96e40efe\") "
Jul 16 11:53:37.127918 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.127891    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b0103f-06a5-4424-81ff-66cb96e40efe-service-ca" (OuterVolumeSpecName: "service-ca") pod "e7b0103f-06a5-4424-81ff-66cb96e40efe" (UID: "e7b0103f-06a5-4424-81ff-66cb96e40efe"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 11:53:37.128055 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.128009    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b0103f-06a5-4424-81ff-66cb96e40efe-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e7b0103f-06a5-4424-81ff-66cb96e40efe" (UID: "e7b0103f-06a5-4424-81ff-66cb96e40efe"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 11:53:37.128159 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.128027    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b0103f-06a5-4424-81ff-66cb96e40efe-console-config" (OuterVolumeSpecName: "console-config") pod "e7b0103f-06a5-4424-81ff-66cb96e40efe" (UID: "e7b0103f-06a5-4424-81ff-66cb96e40efe"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 11:53:37.129938 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.129914    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b0103f-06a5-4424-81ff-66cb96e40efe-kube-api-access-rzwzp" (OuterVolumeSpecName: "kube-api-access-rzwzp") pod "e7b0103f-06a5-4424-81ff-66cb96e40efe" (UID: "e7b0103f-06a5-4424-81ff-66cb96e40efe"). InnerVolumeSpecName "kube-api-access-rzwzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 11:53:37.130058 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.129958    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7b0103f-06a5-4424-81ff-66cb96e40efe-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e7b0103f-06a5-4424-81ff-66cb96e40efe" (UID: "e7b0103f-06a5-4424-81ff-66cb96e40efe"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 11:53:37.130058 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.129968    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7b0103f-06a5-4424-81ff-66cb96e40efe-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e7b0103f-06a5-4424-81ff-66cb96e40efe" (UID: "e7b0103f-06a5-4424-81ff-66cb96e40efe"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 11:53:37.228267 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.228214    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzwzp\" (UniqueName: \"kubernetes.io/projected/e7b0103f-06a5-4424-81ff-66cb96e40efe-kube-api-access-rzwzp\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 11:53:37.228267 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.228236    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7b0103f-06a5-4424-81ff-66cb96e40efe-console-serving-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 11:53:37.228267 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.228245    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e7b0103f-06a5-4424-81ff-66cb96e40efe-oauth-serving-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 11:53:37.228267 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.228254    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e7b0103f-06a5-4424-81ff-66cb96e40efe-console-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 11:53:37.228267 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.228262    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e7b0103f-06a5-4424-81ff-66cb96e40efe-console-oauth-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 11:53:37.228267 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.228270    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b0103f-06a5-4424-81ff-66cb96e40efe-service-ca\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 11:53:37.955925 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.955896    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b699464c7-xztxx"
Jul 16 11:53:37.976083 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.976057    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b699464c7-xztxx"]
Jul 16 11:53:37.978928 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:37.978905    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b699464c7-xztxx"]
Jul 16 11:53:39.297488 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:53:39.297453    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7b0103f-06a5-4424-81ff-66cb96e40efe" path="/var/lib/kubelet/pods/e7b0103f-06a5-4424-81ff-66cb96e40efe/volumes"
Jul 16 11:57:07.030320 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:07.030287    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-krhgr"]
Jul 16 11:57:07.030746 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:07.030524    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b0103f-06a5-4424-81ff-66cb96e40efe" containerName="console"
Jul 16 11:57:07.030746 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:07.030535    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b0103f-06a5-4424-81ff-66cb96e40efe" containerName="console"
Jul 16 11:57:07.030746 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:07.030589    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7b0103f-06a5-4424-81ff-66cb96e40efe" containerName="console"
Jul 16 11:57:07.032222 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:07.032206    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-krhgr"
Jul 16 11:57:07.034767 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:07.034743    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 16 11:57:07.035689 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:07.035671    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-24kl6\""
Jul 16 11:57:07.035753 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:07.035692    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 16 11:57:07.044888 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:07.044866    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-krhgr"]
Jul 16 11:57:07.203721 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:07.203693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f6p8\" (UniqueName: \"kubernetes.io/projected/b97ab991-6723-434f-9227-ff570eaaffcf-kube-api-access-4f6p8\") pod \"cert-manager-55b4d954d5-krhgr\" (UID: \"b97ab991-6723-434f-9227-ff570eaaffcf\") " pod="cert-manager/cert-manager-55b4d954d5-krhgr"
Jul 16 11:57:07.203846 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:07.203735    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b97ab991-6723-434f-9227-ff570eaaffcf-bound-sa-token\") pod \"cert-manager-55b4d954d5-krhgr\" (UID: \"b97ab991-6723-434f-9227-ff570eaaffcf\") " pod="cert-manager/cert-manager-55b4d954d5-krhgr"
Jul 16 11:57:07.304479 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:07.304452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f6p8\" (UniqueName: \"kubernetes.io/projected/b97ab991-6723-434f-9227-ff570eaaffcf-kube-api-access-4f6p8\") pod \"cert-manager-55b4d954d5-krhgr\" (UID: \"b97ab991-6723-434f-9227-ff570eaaffcf\") " pod="cert-manager/cert-manager-55b4d954d5-krhgr"
Jul 16 11:57:07.304585 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:07.304492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b97ab991-6723-434f-9227-ff570eaaffcf-bound-sa-token\") pod \"cert-manager-55b4d954d5-krhgr\" (UID: \"b97ab991-6723-434f-9227-ff570eaaffcf\") " pod="cert-manager/cert-manager-55b4d954d5-krhgr"
Jul 16 11:57:07.313071 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:07.313023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b97ab991-6723-434f-9227-ff570eaaffcf-bound-sa-token\") pod \"cert-manager-55b4d954d5-krhgr\" (UID: \"b97ab991-6723-434f-9227-ff570eaaffcf\") " pod="cert-manager/cert-manager-55b4d954d5-krhgr"
Jul 16 11:57:07.313379 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:07.313360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f6p8\" (UniqueName: \"kubernetes.io/projected/b97ab991-6723-434f-9227-ff570eaaffcf-kube-api-access-4f6p8\") pod \"cert-manager-55b4d954d5-krhgr\" (UID: \"b97ab991-6723-434f-9227-ff570eaaffcf\") " pod="cert-manager/cert-manager-55b4d954d5-krhgr"
Jul 16 11:57:07.341291 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:07.341268    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-krhgr"
Jul 16 11:57:07.471687 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:07.471654    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-krhgr"]
Jul 16 11:57:07.474542 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:57:07.474512    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb97ab991_6723_434f_9227_ff570eaaffcf.slice/crio-6769e95dc4d6d660f913990edd8416f1d7ef0a28493a04b7eecee5214e0fd788 WatchSource:0}: Error finding container 6769e95dc4d6d660f913990edd8416f1d7ef0a28493a04b7eecee5214e0fd788: Status 404 returned error can't find the container with id 6769e95dc4d6d660f913990edd8416f1d7ef0a28493a04b7eecee5214e0fd788
Jul 16 11:57:07.476530 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:07.476513    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 11:57:07.669920 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:07.669850    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-krhgr" event={"ID":"b97ab991-6723-434f-9227-ff570eaaffcf","Type":"ContainerStarted","Data":"6769e95dc4d6d660f913990edd8416f1d7ef0a28493a04b7eecee5214e0fd788"}
Jul 16 11:57:10.684199 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:10.684110    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-krhgr" event={"ID":"b97ab991-6723-434f-9227-ff570eaaffcf","Type":"ContainerStarted","Data":"6158f409b1621cc0ac70e8672d395d86c36280e0e5a9636556e4f29d93973347"}
Jul 16 11:57:10.701537 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:10.701485    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-krhgr" podStartSLOduration=0.776053256 podStartE2EDuration="3.701466892s" totalImagesPullingTime="2.925413636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:57:10.700433558 +0000 UTC m=+372.001210656" watchObservedRunningTime="2026-07-16 11:57:10.701466892 +0000 UTC m=+372.002243989"
Jul 16 11:57:31.420407 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.420374    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"]
Jul 16 11:57:31.424310 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.424293    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"
Jul 16 11:57:31.436948 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.436926    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 16 11:57:31.437065 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.436926    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 16 11:57:31.437065 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.437013    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zvkj4\""
Jul 16 11:57:31.437193 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.437110    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 16 11:57:31.437245 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.437212    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 16 11:57:31.455881 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.455855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36a058c0-7c4b-4724-badc-788bb9c48bd0-tmp\") pod \"opendatahub-operator-controller-manager-57d7447f8f-75zrk\" (UID: \"36a058c0-7c4b-4724-badc-788bb9c48bd0\") " pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"
Jul 16 11:57:31.455974 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.455886    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36a058c0-7c4b-4724-badc-788bb9c48bd0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57d7447f8f-75zrk\" (UID: \"36a058c0-7c4b-4724-badc-788bb9c48bd0\") " pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"
Jul 16 11:57:31.455974 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.455906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36a058c0-7c4b-4724-badc-788bb9c48bd0-webhook-cert\") pod \"opendatahub-operator-controller-manager-57d7447f8f-75zrk\" (UID: \"36a058c0-7c4b-4724-badc-788bb9c48bd0\") " pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"
Jul 16 11:57:31.455974 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.455929    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36a058c0-7c4b-4724-badc-788bb9c48bd0-manifests\") pod \"opendatahub-operator-controller-manager-57d7447f8f-75zrk\" (UID: \"36a058c0-7c4b-4724-badc-788bb9c48bd0\") " pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"
Jul 16 11:57:31.456148 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.456010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgqzm\" (UniqueName: \"kubernetes.io/projected/36a058c0-7c4b-4724-badc-788bb9c48bd0-kube-api-access-sgqzm\") pod \"opendatahub-operator-controller-manager-57d7447f8f-75zrk\" (UID: \"36a058c0-7c4b-4724-badc-788bb9c48bd0\") " pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"
Jul 16 11:57:31.557421 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.557383    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36a058c0-7c4b-4724-badc-788bb9c48bd0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57d7447f8f-75zrk\" (UID: \"36a058c0-7c4b-4724-badc-788bb9c48bd0\") " pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"
Jul 16 11:57:31.557421 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.557422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36a058c0-7c4b-4724-badc-788bb9c48bd0-webhook-cert\") pod \"opendatahub-operator-controller-manager-57d7447f8f-75zrk\" (UID: \"36a058c0-7c4b-4724-badc-788bb9c48bd0\") " pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"
Jul 16 11:57:31.557612 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.557445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36a058c0-7c4b-4724-badc-788bb9c48bd0-manifests\") pod \"opendatahub-operator-controller-manager-57d7447f8f-75zrk\" (UID: \"36a058c0-7c4b-4724-badc-788bb9c48bd0\") " pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"
Jul 16 11:57:31.557612 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.557570    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgqzm\" (UniqueName: \"kubernetes.io/projected/36a058c0-7c4b-4724-badc-788bb9c48bd0-kube-api-access-sgqzm\") pod \"opendatahub-operator-controller-manager-57d7447f8f-75zrk\" (UID: \"36a058c0-7c4b-4724-badc-788bb9c48bd0\") " pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"
Jul 16 11:57:31.557693 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.557665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36a058c0-7c4b-4724-badc-788bb9c48bd0-tmp\") pod \"opendatahub-operator-controller-manager-57d7447f8f-75zrk\" (UID: \"36a058c0-7c4b-4724-badc-788bb9c48bd0\") " pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"
Jul 16 11:57:31.557797 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.557781    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36a058c0-7c4b-4724-badc-788bb9c48bd0-manifests\") pod \"opendatahub-operator-controller-manager-57d7447f8f-75zrk\" (UID: \"36a058c0-7c4b-4724-badc-788bb9c48bd0\") " pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"
Jul 16 11:57:31.559884 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.559855    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36a058c0-7c4b-4724-badc-788bb9c48bd0-tmp\") pod \"opendatahub-operator-controller-manager-57d7447f8f-75zrk\" (UID: \"36a058c0-7c4b-4724-badc-788bb9c48bd0\") " pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"
Jul 16 11:57:31.560013 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.559918    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36a058c0-7c4b-4724-badc-788bb9c48bd0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57d7447f8f-75zrk\" (UID: \"36a058c0-7c4b-4724-badc-788bb9c48bd0\") " pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"
Jul 16 11:57:31.560013 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.559998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36a058c0-7c4b-4724-badc-788bb9c48bd0-webhook-cert\") pod \"opendatahub-operator-controller-manager-57d7447f8f-75zrk\" (UID: \"36a058c0-7c4b-4724-badc-788bb9c48bd0\") " pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"
Jul 16 11:57:31.577564 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.577542    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgqzm\" (UniqueName: \"kubernetes.io/projected/36a058c0-7c4b-4724-badc-788bb9c48bd0-kube-api-access-sgqzm\") pod \"opendatahub-operator-controller-manager-57d7447f8f-75zrk\" (UID: \"36a058c0-7c4b-4724-badc-788bb9c48bd0\") " pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"
Jul 16 11:57:31.592509 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.592485    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"]
Jul 16 11:57:31.734020 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.733933    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"
Jul 16 11:57:31.884053 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:31.884011    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"]
Jul 16 11:57:31.888919 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:57:31.888892    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a058c0_7c4b_4724_badc_788bb9c48bd0.slice/crio-d331bc15f220cff568b9593bda49204227419f8c98658f638c602384c0b7fee5 WatchSource:0}: Error finding container d331bc15f220cff568b9593bda49204227419f8c98658f638c602384c0b7fee5: Status 404 returned error can't find the container with id d331bc15f220cff568b9593bda49204227419f8c98658f638c602384c0b7fee5
Jul 16 11:57:32.760828 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:32.760761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk" event={"ID":"36a058c0-7c4b-4724-badc-788bb9c48bd0","Type":"ContainerStarted","Data":"d331bc15f220cff568b9593bda49204227419f8c98658f638c602384c0b7fee5"}
Jul 16 11:57:35.774532 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:35.774494    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk" event={"ID":"36a058c0-7c4b-4724-badc-788bb9c48bd0","Type":"ContainerStarted","Data":"be8394bf5b32c190ab25d0962b1d5fda4830cfedeb139ce0e109368bee5dfbf9"}
Jul 16 11:57:36.779288 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:36.779251    2578 generic.go:356] "Generic (PLEG): container finished" podID="36a058c0-7c4b-4724-badc-788bb9c48bd0" containerID="be8394bf5b32c190ab25d0962b1d5fda4830cfedeb139ce0e109368bee5dfbf9" exitCode=0
Jul 16 11:57:36.779720 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:36.779345    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk" event={"ID":"36a058c0-7c4b-4724-badc-788bb9c48bd0","Type":"ContainerDied","Data":"be8394bf5b32c190ab25d0962b1d5fda4830cfedeb139ce0e109368bee5dfbf9"}
Jul 16 11:57:37.785293 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:37.785255    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk" event={"ID":"36a058c0-7c4b-4724-badc-788bb9c48bd0","Type":"ContainerStarted","Data":"b9d38dc9438590270ec9c5394fab4631ecb068ba6fad47ea2d8ec0b6eb46c490"}
Jul 16 11:57:37.785737 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:37.785401    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"
Jul 16 11:57:48.790670 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:48.790638    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk"
Jul 16 11:57:48.814816 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:48.814765    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-57d7447f8f-75zrk" podStartSLOduration=13.307865027 podStartE2EDuration="17.814750847s" totalImagesPullingTime="3.446810384s" totalInitContainerRuntime="1.060075436s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:57:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:57:37.83222761 +0000 UTC m=+399.133004709" watchObservedRunningTime="2026-07-16 11:57:48.814750847 +0000 UTC m=+410.115527947"
Jul 16 11:57:52.115030 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.114989    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-69fc9c4ddf-6487s"]
Jul 16 11:57:52.118232 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.118216    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69fc9c4ddf-6487s"
Jul 16 11:57:52.121306 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.121284    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 16 11:57:52.123240 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.123216    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 16 11:57:52.148993 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.148964    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69fc9c4ddf-6487s"]
Jul 16 11:57:52.213076 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.213018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2372c486-6a35-4746-9419-b75cb62ec5cd-tls-certs\") pod \"kube-auth-proxy-69fc9c4ddf-6487s\" (UID: \"2372c486-6a35-4746-9419-b75cb62ec5cd\") " pod="openshift-ingress/kube-auth-proxy-69fc9c4ddf-6487s"
Jul 16 11:57:52.213263 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.213086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2372c486-6a35-4746-9419-b75cb62ec5cd-tmp\") pod \"kube-auth-proxy-69fc9c4ddf-6487s\" (UID: \"2372c486-6a35-4746-9419-b75cb62ec5cd\") " pod="openshift-ingress/kube-auth-proxy-69fc9c4ddf-6487s"
Jul 16 11:57:52.213263 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.213137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twkt6\" (UniqueName: \"kubernetes.io/projected/2372c486-6a35-4746-9419-b75cb62ec5cd-kube-api-access-twkt6\") pod \"kube-auth-proxy-69fc9c4ddf-6487s\" (UID: \"2372c486-6a35-4746-9419-b75cb62ec5cd\") " pod="openshift-ingress/kube-auth-proxy-69fc9c4ddf-6487s"
Jul 16 11:57:52.281571 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.281537    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-95sxr"]
Jul 16 11:57:52.284721 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.284701    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-95sxr"
Jul 16 11:57:52.290367 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.290346    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 16 11:57:52.314170 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.314142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2372c486-6a35-4746-9419-b75cb62ec5cd-tls-certs\") pod \"kube-auth-proxy-69fc9c4ddf-6487s\" (UID: \"2372c486-6a35-4746-9419-b75cb62ec5cd\") " pod="openshift-ingress/kube-auth-proxy-69fc9c4ddf-6487s"
Jul 16 11:57:52.314291 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.314176    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2372c486-6a35-4746-9419-b75cb62ec5cd-tmp\") pod \"kube-auth-proxy-69fc9c4ddf-6487s\" (UID: \"2372c486-6a35-4746-9419-b75cb62ec5cd\") " pod="openshift-ingress/kube-auth-proxy-69fc9c4ddf-6487s"
Jul 16 11:57:52.314329 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.314303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twkt6\" (UniqueName: \"kubernetes.io/projected/2372c486-6a35-4746-9419-b75cb62ec5cd-kube-api-access-twkt6\") pod \"kube-auth-proxy-69fc9c4ddf-6487s\" (UID: \"2372c486-6a35-4746-9419-b75cb62ec5cd\") " pod="openshift-ingress/kube-auth-proxy-69fc9c4ddf-6487s"
Jul 16 11:57:52.315187 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.315165    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-pkh5v\""
Jul 16 11:57:52.316408 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.316387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2372c486-6a35-4746-9419-b75cb62ec5cd-tmp\") pod \"kube-auth-proxy-69fc9c4ddf-6487s\" (UID: \"2372c486-6a35-4746-9419-b75cb62ec5cd\") " pod="openshift-ingress/kube-auth-proxy-69fc9c4ddf-6487s"
Jul 16 11:57:52.316490 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.316461    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2372c486-6a35-4746-9419-b75cb62ec5cd-tls-certs\") pod \"kube-auth-proxy-69fc9c4ddf-6487s\" (UID: \"2372c486-6a35-4746-9419-b75cb62ec5cd\") " pod="openshift-ingress/kube-auth-proxy-69fc9c4ddf-6487s"
Jul 16 11:57:52.323979 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.323956    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-95sxr"]
Jul 16 11:57:52.339676 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.339655    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twkt6\" (UniqueName: \"kubernetes.io/projected/2372c486-6a35-4746-9419-b75cb62ec5cd-kube-api-access-twkt6\") pod \"kube-auth-proxy-69fc9c4ddf-6487s\" (UID: \"2372c486-6a35-4746-9419-b75cb62ec5cd\") " pod="openshift-ingress/kube-auth-proxy-69fc9c4ddf-6487s"
Jul 16 11:57:52.415412 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.415326    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6trdf\" (UniqueName: \"kubernetes.io/projected/36092f20-8fe5-4513-aa16-b10fcaaeed44-kube-api-access-6trdf\") pod \"odh-model-controller-7998c67dc4-95sxr\" (UID: \"36092f20-8fe5-4513-aa16-b10fcaaeed44\") " pod="opendatahub/odh-model-controller-7998c67dc4-95sxr"
Jul 16 11:57:52.415574 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.415430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36092f20-8fe5-4513-aa16-b10fcaaeed44-cert\") pod \"odh-model-controller-7998c67dc4-95sxr\" (UID: \"36092f20-8fe5-4513-aa16-b10fcaaeed44\") " pod="opendatahub/odh-model-controller-7998c67dc4-95sxr"
Jul 16 11:57:52.426878 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.426849    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69fc9c4ddf-6487s"
Jul 16 11:57:52.516522 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.516492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36092f20-8fe5-4513-aa16-b10fcaaeed44-cert\") pod \"odh-model-controller-7998c67dc4-95sxr\" (UID: \"36092f20-8fe5-4513-aa16-b10fcaaeed44\") " pod="opendatahub/odh-model-controller-7998c67dc4-95sxr"
Jul 16 11:57:52.516641 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.516545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6trdf\" (UniqueName: \"kubernetes.io/projected/36092f20-8fe5-4513-aa16-b10fcaaeed44-kube-api-access-6trdf\") pod \"odh-model-controller-7998c67dc4-95sxr\" (UID: \"36092f20-8fe5-4513-aa16-b10fcaaeed44\") " pod="opendatahub/odh-model-controller-7998c67dc4-95sxr"
Jul 16 11:57:52.516692 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:57:52.516670    2578 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 16 11:57:52.516764 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:57:52.516751    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36092f20-8fe5-4513-aa16-b10fcaaeed44-cert podName:36092f20-8fe5-4513-aa16-b10fcaaeed44 nodeName:}" failed. No retries permitted until 2026-07-16 11:57:53.016727919 +0000 UTC m=+414.317504998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/36092f20-8fe5-4513-aa16-b10fcaaeed44-cert") pod "odh-model-controller-7998c67dc4-95sxr" (UID: "36092f20-8fe5-4513-aa16-b10fcaaeed44") : secret "odh-model-controller-webhook-cert" not found
Jul 16 11:57:52.528820 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.528788    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6trdf\" (UniqueName: \"kubernetes.io/projected/36092f20-8fe5-4513-aa16-b10fcaaeed44-kube-api-access-6trdf\") pod \"odh-model-controller-7998c67dc4-95sxr\" (UID: \"36092f20-8fe5-4513-aa16-b10fcaaeed44\") " pod="opendatahub/odh-model-controller-7998c67dc4-95sxr"
Jul 16 11:57:52.588624 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.588596    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69fc9c4ddf-6487s"]
Jul 16 11:57:52.595217 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:57:52.595180    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2372c486_6a35_4746_9419_b75cb62ec5cd.slice/crio-1b0f375bc3af0ad979b15093b5790238fef3128886782ff4fde9f31f5ce8cccc WatchSource:0}: Error finding container 1b0f375bc3af0ad979b15093b5790238fef3128886782ff4fde9f31f5ce8cccc: Status 404 returned error can't find the container with id 1b0f375bc3af0ad979b15093b5790238fef3128886782ff4fde9f31f5ce8cccc
Jul 16 11:57:52.844337 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:52.844305    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69fc9c4ddf-6487s" event={"ID":"2372c486-6a35-4746-9419-b75cb62ec5cd","Type":"ContainerStarted","Data":"1b0f375bc3af0ad979b15093b5790238fef3128886782ff4fde9f31f5ce8cccc"}
Jul 16 11:57:53.021241 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:53.021207    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36092f20-8fe5-4513-aa16-b10fcaaeed44-cert\") pod \"odh-model-controller-7998c67dc4-95sxr\" (UID: \"36092f20-8fe5-4513-aa16-b10fcaaeed44\") " pod="opendatahub/odh-model-controller-7998c67dc4-95sxr"
Jul 16 11:57:53.023598 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:53.023572    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36092f20-8fe5-4513-aa16-b10fcaaeed44-cert\") pod \"odh-model-controller-7998c67dc4-95sxr\" (UID: \"36092f20-8fe5-4513-aa16-b10fcaaeed44\") " pod="opendatahub/odh-model-controller-7998c67dc4-95sxr"
Jul 16 11:57:53.194649 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:53.194581    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-95sxr"
Jul 16 11:57:53.554742 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:53.554703    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-95sxr"]
Jul 16 11:57:53.559226 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:57:53.559196    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36092f20_8fe5_4513_aa16_b10fcaaeed44.slice/crio-6dfab2b0453f8bbfe4ca22ea8928489dc84a1141b0538553f00176b3a5e77214 WatchSource:0}: Error finding container 6dfab2b0453f8bbfe4ca22ea8928489dc84a1141b0538553f00176b3a5e77214: Status 404 returned error can't find the container with id 6dfab2b0453f8bbfe4ca22ea8928489dc84a1141b0538553f00176b3a5e77214
Jul 16 11:57:53.849665 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:53.849587    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-95sxr" event={"ID":"36092f20-8fe5-4513-aa16-b10fcaaeed44","Type":"ContainerStarted","Data":"6dfab2b0453f8bbfe4ca22ea8928489dc84a1141b0538553f00176b3a5e77214"}
Jul 16 11:57:54.692711 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.692354    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"]
Jul 16 11:57:54.696198 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.696121    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.700231 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.700206    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 16 11:57:54.700362 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.700245    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 16 11:57:54.700445 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.700206    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 16 11:57:54.701726 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.700741    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jn9q7\""
Jul 16 11:57:54.701726 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.700923    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 16 11:57:54.713014 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.712989    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"]
Jul 16 11:57:54.837320 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.837282    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3ee66c10-9145-4581-9bf8-92cad3591fcd-cacerts\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.837490 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.837337    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.837490 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.837416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.837490 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.837447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ee66c10-9145-4581-9bf8-92cad3591fcd-local-certs\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.837490 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.837475    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-token\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.837712 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.837507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcf7w\" (UniqueName: \"kubernetes.io/projected/3ee66c10-9145-4581-9bf8-92cad3591fcd-kube-api-access-fcf7w\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.837712 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.837626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.938628 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.938588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcf7w\" (UniqueName: \"kubernetes.io/projected/3ee66c10-9145-4581-9bf8-92cad3591fcd-kube-api-access-fcf7w\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.938787 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.938660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.938787 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.938693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3ee66c10-9145-4581-9bf8-92cad3591fcd-cacerts\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.938787 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.938722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.938787 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.938780    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.939002 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.938808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ee66c10-9145-4581-9bf8-92cad3591fcd-local-certs\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.939002 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.938835    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-token\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.939481 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.939428    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.941414 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.941388    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ee66c10-9145-4581-9bf8-92cad3591fcd-local-certs\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.942009 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.941978    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.942140 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.942068    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.942253 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.942213    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3ee66c10-9145-4581-9bf8-92cad3591fcd-cacerts\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.948785 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.948726    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcf7w\" (UniqueName: \"kubernetes.io/projected/3ee66c10-9145-4581-9bf8-92cad3591fcd-kube-api-access-fcf7w\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:54.950605 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:54.950583    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-token\") pod \"istiod-openshift-gateway-7574459656-4jsjm\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:55.010982 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:55.010782    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:56.788488 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:56.788458    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"]
Jul 16 11:57:56.790406 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:57:56.790379    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee66c10_9145_4581_9bf8_92cad3591fcd.slice/crio-325f51417a224527b12faf5a3003c6441eedfdaff37a525c5c529fd26a9de3f2 WatchSource:0}: Error finding container 325f51417a224527b12faf5a3003c6441eedfdaff37a525c5c529fd26a9de3f2: Status 404 returned error can't find the container with id 325f51417a224527b12faf5a3003c6441eedfdaff37a525c5c529fd26a9de3f2
Jul 16 11:57:56.863825 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:56.863748    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm" event={"ID":"3ee66c10-9145-4581-9bf8-92cad3591fcd","Type":"ContainerStarted","Data":"325f51417a224527b12faf5a3003c6441eedfdaff37a525c5c529fd26a9de3f2"}
Jul 16 11:57:56.865313 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:56.865276    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-95sxr" event={"ID":"36092f20-8fe5-4513-aa16-b10fcaaeed44","Type":"ContainerStarted","Data":"4a54620e7335b61775da15de85c6ad3f143b0fdba86e0c0c126917dec601f138"}
Jul 16 11:57:56.865453 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:56.865341    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-95sxr"
Jul 16 11:57:56.866997 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:56.866973    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69fc9c4ddf-6487s" event={"ID":"2372c486-6a35-4746-9419-b75cb62ec5cd","Type":"ContainerStarted","Data":"bd6b1e5b20f5ba95dcd55cda035af4309e06bf0307489c476b9fbab3494b8fa3"}
Jul 16 11:57:56.970739 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:56.970691    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-69fc9c4ddf-6487s" podStartSLOduration=0.983028836 podStartE2EDuration="4.970674238s" totalImagesPullingTime="3.987645402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:57:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:57:56.970480823 +0000 UTC m=+418.271257922" watchObservedRunningTime="2026-07-16 11:57:56.970674238 +0000 UTC m=+418.271451335"
Jul 16 11:57:56.971679 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:56.971648    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-95sxr" podStartSLOduration=1.8876386539999999 podStartE2EDuration="4.971635082s" totalImagesPullingTime="3.083996428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:57:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:57:56.922847824 +0000 UTC m=+418.223624921" watchObservedRunningTime="2026-07-16 11:57:56.971635082 +0000 UTC m=+418.272412180"
Jul 16 11:57:59.582787 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:59.582740    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 11:57:59.583061 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:59.582836    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 11:57:59.884240 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:59.884151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm" event={"ID":"3ee66c10-9145-4581-9bf8-92cad3591fcd","Type":"ContainerStarted","Data":"de0c441824a120e91ae556c1e593c09a3dfce1ec59ae673e7860fe021c1677fc"}
Jul 16 11:57:59.884240 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:59.884214    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:57:59.967865 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:57:59.967812    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm" podStartSLOduration=3.177810536 podStartE2EDuration="5.96779609s" totalImagesPullingTime="2.789985554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:57:59.963081954 +0000 UTC m=+421.263859069" watchObservedRunningTime="2026-07-16 11:57:59.96779609 +0000 UTC m=+421.268573189"
Jul 16 11:58:00.889525 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:00.889487    2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-4jsjm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 11:58:00.889945 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:00.889549    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm" podUID="3ee66c10-9145-4581-9bf8-92cad3591fcd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 11:58:01.162632 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.162556    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"]
Jul 16 11:58:01.165935 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.165913    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.173258 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.173236    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-hstjv\""
Jul 16 11:58:01.196625 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.196599    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"]
Jul 16 11:58:01.295923 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.295900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.296066 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.295930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.296066 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.295950    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.296066 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.295996    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.296066 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.296016    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.296066 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.296047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.296262 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.296133    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.296262 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.296158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.296262 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.296218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2dgh\" (UniqueName: \"kubernetes.io/projected/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-kube-api-access-d2dgh\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.397342 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.397312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.397465 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.397350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.397465 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.397377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.397465 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.397411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.397465 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.397440    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.397657 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.397468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.397657 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.397500    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.397756 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.397716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.397809 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.397792    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2dgh\" (UniqueName: \"kubernetes.io/projected/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-kube-api-access-d2dgh\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.397867 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.397815    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.397919 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.397879    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.398091 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.398071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.398201 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.398184    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.398273 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.398258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.400263 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.400237    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.400439 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.400422    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.406448 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.406426    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.413591 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.413531    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2dgh\" (UniqueName: \"kubernetes.io/projected/52d7bf4e-b9d9-4b85-a599-8976e5cddd3a-kube-api-access-d2dgh\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785nq4x\" (UID: \"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.476266 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.476244    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:01.677233 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.677156    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"]
Jul 16 11:58:01.683810 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:58:01.683780    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d7bf4e_b9d9_4b85_a599_8976e5cddd3a.slice/crio-31524f29c75d6f0913e66d7bdf7468a910f56b3440fdfa2df12b8d2091f34632 WatchSource:0}: Error finding container 31524f29c75d6f0913e66d7bdf7468a910f56b3440fdfa2df12b8d2091f34632: Status 404 returned error can't find the container with id 31524f29c75d6f0913e66d7bdf7468a910f56b3440fdfa2df12b8d2091f34632
Jul 16 11:58:01.892465 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:01.892430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x" event={"ID":"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a","Type":"ContainerStarted","Data":"31524f29c75d6f0913e66d7bdf7468a910f56b3440fdfa2df12b8d2091f34632"}
Jul 16 11:58:03.889414 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:03.889381    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:58:04.348307 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:04.348270    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 11:58:04.348414 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:04.348348    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 11:58:04.348414 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:04.348376    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 11:58:04.907844 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:04.907811    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x" event={"ID":"52d7bf4e-b9d9-4b85-a599-8976e5cddd3a","Type":"ContainerStarted","Data":"0e19760b8080d06d6962c5774f4ff477e69ec3f04807c100a47dbb3bdd4296be"}
Jul 16 11:58:05.392488 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.392424    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x" podStartSLOduration=1.730204465 podStartE2EDuration="4.392402484s" totalImagesPullingTime="2.662198019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:58:04.96922618 +0000 UTC m=+426.270003279" watchObservedRunningTime="2026-07-16 11:58:05.392402484 +0000 UTC m=+426.693179583"
Jul 16 11:58:05.393542 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.393523    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-qqwbh"]
Jul 16 11:58:05.396893 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.396854    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-qqwbh"
Jul 16 11:58:05.399832 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.399810    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-ffchh\""
Jul 16 11:58:05.400167 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.400148    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 16 11:58:05.418518 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.418490    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-c99d7"]
Jul 16 11:58:05.421925 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.421907    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-qqwbh"]
Jul 16 11:58:05.422023 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.422005    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c99d7"
Jul 16 11:58:05.425354 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.425331    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-rff6x\""
Jul 16 11:58:05.425446 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.425433    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 16 11:58:05.438933 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.438911    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-c99d7"]
Jul 16 11:58:05.476545 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.476517    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:05.481177 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.481157    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:05.535300 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.535270    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v2nn\" (UniqueName: \"kubernetes.io/projected/4973c159-ec99-42a4-9114-388ef2c6036a-kube-api-access-6v2nn\") pod \"kserve-controller-manager-d6f4b65b8-qqwbh\" (UID: \"4973c159-ec99-42a4-9114-388ef2c6036a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-qqwbh"
Jul 16 11:58:05.535407 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.535329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mzrs\" (UniqueName: \"kubernetes.io/projected/78107917-5ba9-4339-bd60-d0ee4ab408b4-kube-api-access-9mzrs\") pod \"llmisvc-controller-manager-777c6bd7d5-c99d7\" (UID: \"78107917-5ba9-4339-bd60-d0ee4ab408b4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c99d7"
Jul 16 11:58:05.535407 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.535392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4973c159-ec99-42a4-9114-388ef2c6036a-cert\") pod \"kserve-controller-manager-d6f4b65b8-qqwbh\" (UID: \"4973c159-ec99-42a4-9114-388ef2c6036a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-qqwbh"
Jul 16 11:58:05.535482 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.535469    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78107917-5ba9-4339-bd60-d0ee4ab408b4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-c99d7\" (UID: \"78107917-5ba9-4339-bd60-d0ee4ab408b4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c99d7"
Jul 16 11:58:05.635911 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.635885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mzrs\" (UniqueName: \"kubernetes.io/projected/78107917-5ba9-4339-bd60-d0ee4ab408b4-kube-api-access-9mzrs\") pod \"llmisvc-controller-manager-777c6bd7d5-c99d7\" (UID: \"78107917-5ba9-4339-bd60-d0ee4ab408b4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c99d7"
Jul 16 11:58:05.636020 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.635919    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4973c159-ec99-42a4-9114-388ef2c6036a-cert\") pod \"kserve-controller-manager-d6f4b65b8-qqwbh\" (UID: \"4973c159-ec99-42a4-9114-388ef2c6036a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-qqwbh"
Jul 16 11:58:05.636020 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.635947    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78107917-5ba9-4339-bd60-d0ee4ab408b4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-c99d7\" (UID: \"78107917-5ba9-4339-bd60-d0ee4ab408b4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c99d7"
Jul 16 11:58:05.636020 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.635980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v2nn\" (UniqueName: \"kubernetes.io/projected/4973c159-ec99-42a4-9114-388ef2c6036a-kube-api-access-6v2nn\") pod \"kserve-controller-manager-d6f4b65b8-qqwbh\" (UID: \"4973c159-ec99-42a4-9114-388ef2c6036a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-qqwbh"
Jul 16 11:58:05.638381 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.638361    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78107917-5ba9-4339-bd60-d0ee4ab408b4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-c99d7\" (UID: \"78107917-5ba9-4339-bd60-d0ee4ab408b4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c99d7"
Jul 16 11:58:05.638475 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.638424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4973c159-ec99-42a4-9114-388ef2c6036a-cert\") pod \"kserve-controller-manager-d6f4b65b8-qqwbh\" (UID: \"4973c159-ec99-42a4-9114-388ef2c6036a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-qqwbh"
Jul 16 11:58:05.661336 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.661276    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v2nn\" (UniqueName: \"kubernetes.io/projected/4973c159-ec99-42a4-9114-388ef2c6036a-kube-api-access-6v2nn\") pod \"kserve-controller-manager-d6f4b65b8-qqwbh\" (UID: \"4973c159-ec99-42a4-9114-388ef2c6036a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-qqwbh"
Jul 16 11:58:05.667191 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.667165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mzrs\" (UniqueName: \"kubernetes.io/projected/78107917-5ba9-4339-bd60-d0ee4ab408b4-kube-api-access-9mzrs\") pod \"llmisvc-controller-manager-777c6bd7d5-c99d7\" (UID: \"78107917-5ba9-4339-bd60-d0ee4ab408b4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c99d7"
Jul 16 11:58:05.707169 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.707146    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-qqwbh"
Jul 16 11:58:05.731261 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.731242    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c99d7"
Jul 16 11:58:05.867775 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:58:05.867732    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4973c159_ec99_42a4_9114_388ef2c6036a.slice/crio-a8fc0dbd1578fd05edfaf15e5caafc2d69796a81ab7b48be83e7c18fde88ba67 WatchSource:0}: Error finding container a8fc0dbd1578fd05edfaf15e5caafc2d69796a81ab7b48be83e7c18fde88ba67: Status 404 returned error can't find the container with id a8fc0dbd1578fd05edfaf15e5caafc2d69796a81ab7b48be83e7c18fde88ba67
Jul 16 11:58:05.870299 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.870276    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-qqwbh"]
Jul 16 11:58:05.913116 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.912958    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-c99d7"]
Jul 16 11:58:05.913887 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.913862    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-qqwbh" event={"ID":"4973c159-ec99-42a4-9114-388ef2c6036a","Type":"ContainerStarted","Data":"a8fc0dbd1578fd05edfaf15e5caafc2d69796a81ab7b48be83e7c18fde88ba67"}
Jul 16 11:58:05.914169 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.914151    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:05.915133 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:05.915114    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785nq4x"
Jul 16 11:58:05.915558 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:58:05.915538    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod78107917_5ba9_4339_bd60_d0ee4ab408b4.slice/crio-00cd739b6c09638c7c16569232e27d34dbfd25008d00c5d8981f252c33848c96 WatchSource:0}: Error finding container 00cd739b6c09638c7c16569232e27d34dbfd25008d00c5d8981f252c33848c96: Status 404 returned error can't find the container with id 00cd739b6c09638c7c16569232e27d34dbfd25008d00c5d8981f252c33848c96
Jul 16 11:58:06.918729 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:06.918696    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c99d7" event={"ID":"78107917-5ba9-4339-bd60-d0ee4ab408b4","Type":"ContainerStarted","Data":"00cd739b6c09638c7c16569232e27d34dbfd25008d00c5d8981f252c33848c96"}
Jul 16 11:58:07.389693 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.389663    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"]
Jul 16 11:58:07.389858 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.389839    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm" podUID="3ee66c10-9145-4581-9bf8-92cad3591fcd" containerName="discovery" containerID="cri-o://de0c441824a120e91ae556c1e593c09a3dfce1ec59ae673e7860fe021c1677fc" gracePeriod=30
Jul 16 11:58:07.657898 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.657869    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:58:07.756029 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.755998    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-csr-dns-cert\") pod \"3ee66c10-9145-4581-9bf8-92cad3591fcd\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") "
Jul 16 11:58:07.756196 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.756144    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-csr-ca-configmap\") pod \"3ee66c10-9145-4581-9bf8-92cad3591fcd\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") "
Jul 16 11:58:07.756272 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.756203    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcf7w\" (UniqueName: \"kubernetes.io/projected/3ee66c10-9145-4581-9bf8-92cad3591fcd-kube-api-access-fcf7w\") pod \"3ee66c10-9145-4581-9bf8-92cad3591fcd\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") "
Jul 16 11:58:07.756430 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.756401    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ee66c10-9145-4581-9bf8-92cad3591fcd-local-certs\") pod \"3ee66c10-9145-4581-9bf8-92cad3591fcd\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") "
Jul 16 11:58:07.756561 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.756474    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3ee66c10-9145-4581-9bf8-92cad3591fcd-cacerts\") pod \"3ee66c10-9145-4581-9bf8-92cad3591fcd\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") "
Jul 16 11:58:07.756561 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.756507    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-token\") pod \"3ee66c10-9145-4581-9bf8-92cad3591fcd\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") "
Jul 16 11:58:07.756561 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.756539    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "3ee66c10-9145-4581-9bf8-92cad3591fcd" (UID: "3ee66c10-9145-4581-9bf8-92cad3591fcd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 11:58:07.756718 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.756548    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-kubeconfig\") pod \"3ee66c10-9145-4581-9bf8-92cad3591fcd\" (UID: \"3ee66c10-9145-4581-9bf8-92cad3591fcd\") "
Jul 16 11:58:07.756880 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.756850    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-csr-ca-configmap\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 11:58:07.758834 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.758774    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "3ee66c10-9145-4581-9bf8-92cad3591fcd" (UID: "3ee66c10-9145-4581-9bf8-92cad3591fcd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 11:58:07.759265 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.759026    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-token" (OuterVolumeSpecName: "istio-token") pod "3ee66c10-9145-4581-9bf8-92cad3591fcd" (UID: "3ee66c10-9145-4581-9bf8-92cad3591fcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 11:58:07.759265 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.759151    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "3ee66c10-9145-4581-9bf8-92cad3591fcd" (UID: "3ee66c10-9145-4581-9bf8-92cad3591fcd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 11:58:07.759265 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.759205    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee66c10-9145-4581-9bf8-92cad3591fcd-kube-api-access-fcf7w" (OuterVolumeSpecName: "kube-api-access-fcf7w") pod "3ee66c10-9145-4581-9bf8-92cad3591fcd" (UID: "3ee66c10-9145-4581-9bf8-92cad3591fcd"). InnerVolumeSpecName "kube-api-access-fcf7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 11:58:07.759460 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.759290    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee66c10-9145-4581-9bf8-92cad3591fcd-local-certs" (OuterVolumeSpecName: "local-certs") pod "3ee66c10-9145-4581-9bf8-92cad3591fcd" (UID: "3ee66c10-9145-4581-9bf8-92cad3591fcd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 11:58:07.759460 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.759345    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ee66c10-9145-4581-9bf8-92cad3591fcd-cacerts" (OuterVolumeSpecName: "cacerts") pod "3ee66c10-9145-4581-9bf8-92cad3591fcd" (UID: "3ee66c10-9145-4581-9bf8-92cad3591fcd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 11:58:07.858083 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.858020    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-csr-dns-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 11:58:07.858083 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.858081    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcf7w\" (UniqueName: \"kubernetes.io/projected/3ee66c10-9145-4581-9bf8-92cad3591fcd-kube-api-access-fcf7w\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 11:58:07.858320 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.858102    2578 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ee66c10-9145-4581-9bf8-92cad3591fcd-local-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 11:58:07.858320 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.858118    2578 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3ee66c10-9145-4581-9bf8-92cad3591fcd-cacerts\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 11:58:07.858320 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.858131    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 11:58:07.858320 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.858144    2578 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ee66c10-9145-4581-9bf8-92cad3591fcd-istio-kubeconfig\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 11:58:07.873679 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.873140    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-95sxr"
Jul 16 11:58:07.924984 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.924917    2578 generic.go:356] "Generic (PLEG): container finished" podID="3ee66c10-9145-4581-9bf8-92cad3591fcd" containerID="de0c441824a120e91ae556c1e593c09a3dfce1ec59ae673e7860fe021c1677fc" exitCode=0
Jul 16 11:58:07.924984 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.924962    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"
Jul 16 11:58:07.925481 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.924997    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm" event={"ID":"3ee66c10-9145-4581-9bf8-92cad3591fcd","Type":"ContainerDied","Data":"de0c441824a120e91ae556c1e593c09a3dfce1ec59ae673e7860fe021c1677fc"}
Jul 16 11:58:07.925481 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.925057    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm" event={"ID":"3ee66c10-9145-4581-9bf8-92cad3591fcd","Type":"ContainerDied","Data":"325f51417a224527b12faf5a3003c6441eedfdaff37a525c5c529fd26a9de3f2"}
Jul 16 11:58:07.925481 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.925079    2578 scope.go:117] "RemoveContainer" containerID="de0c441824a120e91ae556c1e593c09a3dfce1ec59ae673e7860fe021c1677fc"
Jul 16 11:58:07.934279 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.934261    2578 scope.go:117] "RemoveContainer" containerID="de0c441824a120e91ae556c1e593c09a3dfce1ec59ae673e7860fe021c1677fc"
Jul 16 11:58:07.934571 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:58:07.934551    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de0c441824a120e91ae556c1e593c09a3dfce1ec59ae673e7860fe021c1677fc\": container with ID starting with de0c441824a120e91ae556c1e593c09a3dfce1ec59ae673e7860fe021c1677fc not found: ID does not exist" containerID="de0c441824a120e91ae556c1e593c09a3dfce1ec59ae673e7860fe021c1677fc"
Jul 16 11:58:07.934619 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.934580    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de0c441824a120e91ae556c1e593c09a3dfce1ec59ae673e7860fe021c1677fc"} err="failed to get container status \"de0c441824a120e91ae556c1e593c09a3dfce1ec59ae673e7860fe021c1677fc\": rpc error: code = NotFound desc = could not find container \"de0c441824a120e91ae556c1e593c09a3dfce1ec59ae673e7860fe021c1677fc\": container with ID starting with de0c441824a120e91ae556c1e593c09a3dfce1ec59ae673e7860fe021c1677fc not found: ID does not exist"
Jul 16 11:58:07.969935 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.969899    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"]
Jul 16 11:58:07.975989 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:07.975935    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4jsjm"]
Jul 16 11:58:09.298987 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:09.298957    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ee66c10-9145-4581-9bf8-92cad3591fcd" path="/var/lib/kubelet/pods/3ee66c10-9145-4581-9bf8-92cad3591fcd/volumes"
Jul 16 11:58:09.943788 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:09.943755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c99d7" event={"ID":"78107917-5ba9-4339-bd60-d0ee4ab408b4","Type":"ContainerStarted","Data":"93e938c17f6abcc8963e3e9ec0a2b9982b0afba9943ff25fc8c1fe3fa61a8fce"}
Jul 16 11:58:09.943971 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:09.943900    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c99d7"
Jul 16 11:58:09.945338 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:09.945312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-qqwbh" event={"ID":"4973c159-ec99-42a4-9114-388ef2c6036a","Type":"ContainerStarted","Data":"c43b19cf9c85866a3dce040b7ad7a48252ed6dbcc053c20c7f2f9532fc8ec336"}
Jul 16 11:58:09.945464 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:09.945371    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-qqwbh"
Jul 16 11:58:09.989179 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:09.989133    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c99d7" podStartSLOduration=1.710687893 podStartE2EDuration="4.989119031s" totalImagesPullingTime="3.278431138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:58:09.988907319 +0000 UTC m=+431.289684417" watchObservedRunningTime="2026-07-16 11:58:09.989119031 +0000 UTC m=+431.289896128"
Jul 16 11:58:40.952082 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:40.951982    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c99d7"
Jul 16 11:58:40.955151 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:40.955127    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-qqwbh"
Jul 16 11:58:40.972497 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:40.972452    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-qqwbh" podStartSLOduration=32.649065301 podStartE2EDuration="35.972438128s" totalImagesPullingTime="3.323372827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:58:10.039610438 +0000 UTC m=+431.340387540" watchObservedRunningTime="2026-07-16 11:58:40.972438128 +0000 UTC m=+462.273215228"
Jul 16 11:58:53.016294 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.016262    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fd2v4"]
Jul 16 11:58:53.016703 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.016556    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ee66c10-9145-4581-9bf8-92cad3591fcd" containerName="discovery"
Jul 16 11:58:53.016703 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.016568    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ee66c10-9145-4581-9bf8-92cad3591fcd" containerName="discovery"
Jul 16 11:58:53.016703 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.016631    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ee66c10-9145-4581-9bf8-92cad3591fcd" containerName="discovery"
Jul 16 11:58:53.018516 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.018501    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fd2v4"
Jul 16 11:58:53.022140 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.022121    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-79c98\""
Jul 16 11:58:53.023074 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.023057    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 11:58:53.025465 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.025447    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 11:58:53.091829 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.091806    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fd2v4"]
Jul 16 11:58:53.107457 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.107429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw75b\" (UniqueName: \"kubernetes.io/projected/8ffa5de0-7bd7-477e-a207-36642b540678-kube-api-access-dw75b\") pod \"kuadrant-operator-catalog-fd2v4\" (UID: \"8ffa5de0-7bd7-477e-a207-36642b540678\") " pod="kuadrant-system/kuadrant-operator-catalog-fd2v4"
Jul 16 11:58:53.208680 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.208653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw75b\" (UniqueName: \"kubernetes.io/projected/8ffa5de0-7bd7-477e-a207-36642b540678-kube-api-access-dw75b\") pod \"kuadrant-operator-catalog-fd2v4\" (UID: \"8ffa5de0-7bd7-477e-a207-36642b540678\") " pod="kuadrant-system/kuadrant-operator-catalog-fd2v4"
Jul 16 11:58:53.223683 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.223651    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw75b\" (UniqueName: \"kubernetes.io/projected/8ffa5de0-7bd7-477e-a207-36642b540678-kube-api-access-dw75b\") pod \"kuadrant-operator-catalog-fd2v4\" (UID: \"8ffa5de0-7bd7-477e-a207-36642b540678\") " pod="kuadrant-system/kuadrant-operator-catalog-fd2v4"
Jul 16 11:58:53.283573 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.283516    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fd2v4"]
Jul 16 11:58:53.283747 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.283736    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fd2v4"
Jul 16 11:58:53.430788 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.430763    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fd2v4"]
Jul 16 11:58:53.434364 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:58:53.434340    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ffa5de0_7bd7_477e_a207_36642b540678.slice/crio-907626bf2666f7ed936aa2a46b60c553f07e4268beb63768bf4dba036ad6f8e0 WatchSource:0}: Error finding container 907626bf2666f7ed936aa2a46b60c553f07e4268beb63768bf4dba036ad6f8e0: Status 404 returned error can't find the container with id 907626bf2666f7ed936aa2a46b60c553f07e4268beb63768bf4dba036ad6f8e0
Jul 16 11:58:53.471785 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.471763    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gb4pb"]
Jul 16 11:58:53.474413 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.474398    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gb4pb"
Jul 16 11:58:53.489184 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.489163    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gb4pb"]
Jul 16 11:58:53.511988 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.511968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6p2q\" (UniqueName: \"kubernetes.io/projected/3c23f979-c73c-4424-916e-cf0229186561-kube-api-access-c6p2q\") pod \"kuadrant-operator-catalog-gb4pb\" (UID: \"3c23f979-c73c-4424-916e-cf0229186561\") " pod="kuadrant-system/kuadrant-operator-catalog-gb4pb"
Jul 16 11:58:53.612514 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.612493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6p2q\" (UniqueName: \"kubernetes.io/projected/3c23f979-c73c-4424-916e-cf0229186561-kube-api-access-c6p2q\") pod \"kuadrant-operator-catalog-gb4pb\" (UID: \"3c23f979-c73c-4424-916e-cf0229186561\") " pod="kuadrant-system/kuadrant-operator-catalog-gb4pb"
Jul 16 11:58:53.623679 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.623660    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6p2q\" (UniqueName: \"kubernetes.io/projected/3c23f979-c73c-4424-916e-cf0229186561-kube-api-access-c6p2q\") pod \"kuadrant-operator-catalog-gb4pb\" (UID: \"3c23f979-c73c-4424-916e-cf0229186561\") " pod="kuadrant-system/kuadrant-operator-catalog-gb4pb"
Jul 16 11:58:53.783419 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.783392    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gb4pb"
Jul 16 11:58:53.919565 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:53.919543    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gb4pb"]
Jul 16 11:58:53.957875 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:58:53.957842    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c23f979_c73c_4424_916e_cf0229186561.slice/crio-1ba16bf09d69f23fa061ffe344b1848edf8d9d3b711b5485c8877aacf5837fa4 WatchSource:0}: Error finding container 1ba16bf09d69f23fa061ffe344b1848edf8d9d3b711b5485c8877aacf5837fa4: Status 404 returned error can't find the container with id 1ba16bf09d69f23fa061ffe344b1848edf8d9d3b711b5485c8877aacf5837fa4
Jul 16 11:58:54.125082 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:54.125050    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fd2v4" event={"ID":"8ffa5de0-7bd7-477e-a207-36642b540678","Type":"ContainerStarted","Data":"907626bf2666f7ed936aa2a46b60c553f07e4268beb63768bf4dba036ad6f8e0"}
Jul 16 11:58:54.126140 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:54.126114    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gb4pb" event={"ID":"3c23f979-c73c-4424-916e-cf0229186561","Type":"ContainerStarted","Data":"1ba16bf09d69f23fa061ffe344b1848edf8d9d3b711b5485c8877aacf5837fa4"}
Jul 16 11:58:56.137221 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:56.137179    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fd2v4" event={"ID":"8ffa5de0-7bd7-477e-a207-36642b540678","Type":"ContainerStarted","Data":"50c9f1d1824a096d33f27d8805a8ca60d114c88edb5e5f4c0b3ed0fc431a3000"}
Jul 16 11:58:56.137687 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:56.137260    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-fd2v4" podUID="8ffa5de0-7bd7-477e-a207-36642b540678" containerName="registry-server" containerID="cri-o://50c9f1d1824a096d33f27d8805a8ca60d114c88edb5e5f4c0b3ed0fc431a3000" gracePeriod=2
Jul 16 11:58:56.138664 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:56.138638    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gb4pb" event={"ID":"3c23f979-c73c-4424-916e-cf0229186561","Type":"ContainerStarted","Data":"7073fcd4924231a608e889ccf705c4c1709fd4381ecffe25df0644c3eb3a84a0"}
Jul 16 11:58:56.210006 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:56.209953    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-fd2v4" podStartSLOduration=2.218344335 podStartE2EDuration="4.20993688s" totalImagesPullingTime="1.991592545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:58:56.18002423 +0000 UTC m=+477.480801328" watchObservedRunningTime="2026-07-16 11:58:56.20993688 +0000 UTC m=+477.510713978"
Jul 16 11:58:56.210290 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:56.210268    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-gb4pb" podStartSLOduration=1.739380937 podStartE2EDuration="3.210262946s" totalImagesPullingTime="1.470882009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:58:56.209416883 +0000 UTC m=+477.510193980" watchObservedRunningTime="2026-07-16 11:58:56.210262946 +0000 UTC m=+477.511040081"
Jul 16 11:58:56.381417 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:56.381389    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fd2v4"
Jul 16 11:58:56.435194 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:56.435122    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw75b\" (UniqueName: \"kubernetes.io/projected/8ffa5de0-7bd7-477e-a207-36642b540678-kube-api-access-dw75b\") pod \"8ffa5de0-7bd7-477e-a207-36642b540678\" (UID: \"8ffa5de0-7bd7-477e-a207-36642b540678\") "
Jul 16 11:58:56.437248 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:56.437217    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ffa5de0-7bd7-477e-a207-36642b540678-kube-api-access-dw75b" (OuterVolumeSpecName: "kube-api-access-dw75b") pod "8ffa5de0-7bd7-477e-a207-36642b540678" (UID: "8ffa5de0-7bd7-477e-a207-36642b540678"). InnerVolumeSpecName "kube-api-access-dw75b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 11:58:56.536425 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:56.536386    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw75b\" (UniqueName: \"kubernetes.io/projected/8ffa5de0-7bd7-477e-a207-36642b540678-kube-api-access-dw75b\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 11:58:57.144241 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:57.144207    2578 generic.go:356] "Generic (PLEG): container finished" podID="8ffa5de0-7bd7-477e-a207-36642b540678" containerID="50c9f1d1824a096d33f27d8805a8ca60d114c88edb5e5f4c0b3ed0fc431a3000" exitCode=0
Jul 16 11:58:57.144669 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:57.144247    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fd2v4"
Jul 16 11:58:57.144669 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:57.144296    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fd2v4" event={"ID":"8ffa5de0-7bd7-477e-a207-36642b540678","Type":"ContainerDied","Data":"50c9f1d1824a096d33f27d8805a8ca60d114c88edb5e5f4c0b3ed0fc431a3000"}
Jul 16 11:58:57.144669 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:57.144345    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fd2v4" event={"ID":"8ffa5de0-7bd7-477e-a207-36642b540678","Type":"ContainerDied","Data":"907626bf2666f7ed936aa2a46b60c553f07e4268beb63768bf4dba036ad6f8e0"}
Jul 16 11:58:57.144669 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:57.144365    2578 scope.go:117] "RemoveContainer" containerID="50c9f1d1824a096d33f27d8805a8ca60d114c88edb5e5f4c0b3ed0fc431a3000"
Jul 16 11:58:57.153171 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:57.153151    2578 scope.go:117] "RemoveContainer" containerID="50c9f1d1824a096d33f27d8805a8ca60d114c88edb5e5f4c0b3ed0fc431a3000"
Jul 16 11:58:57.153413 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:58:57.153396    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50c9f1d1824a096d33f27d8805a8ca60d114c88edb5e5f4c0b3ed0fc431a3000\": container with ID starting with 50c9f1d1824a096d33f27d8805a8ca60d114c88edb5e5f4c0b3ed0fc431a3000 not found: ID does not exist" containerID="50c9f1d1824a096d33f27d8805a8ca60d114c88edb5e5f4c0b3ed0fc431a3000"
Jul 16 11:58:57.153458 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:57.153423    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50c9f1d1824a096d33f27d8805a8ca60d114c88edb5e5f4c0b3ed0fc431a3000"} err="failed to get container status \"50c9f1d1824a096d33f27d8805a8ca60d114c88edb5e5f4c0b3ed0fc431a3000\": rpc error: code = NotFound desc = could not find container \"50c9f1d1824a096d33f27d8805a8ca60d114c88edb5e5f4c0b3ed0fc431a3000\": container with ID starting with 50c9f1d1824a096d33f27d8805a8ca60d114c88edb5e5f4c0b3ed0fc431a3000 not found: ID does not exist"
Jul 16 11:58:57.199437 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:57.199417    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fd2v4"]
Jul 16 11:58:57.207236 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:57.207215    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fd2v4"]
Jul 16 11:58:57.299374 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:58:57.299344    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ffa5de0-7bd7-477e-a207-36642b540678" path="/var/lib/kubelet/pods/8ffa5de0-7bd7-477e-a207-36642b540678/volumes"
Jul 16 11:59:03.784585 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:03.784552    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-gb4pb"
Jul 16 11:59:03.785060 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:03.784607    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-gb4pb"
Jul 16 11:59:03.806800 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:03.806777    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-gb4pb"
Jul 16 11:59:04.195752 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:04.195728    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-gb4pb"
Jul 16 11:59:27.691633 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:27.691596    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"]
Jul 16 11:59:27.692098 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:27.691898    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ffa5de0-7bd7-477e-a207-36642b540678" containerName="registry-server"
Jul 16 11:59:27.692098 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:27.691909    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ffa5de0-7bd7-477e-a207-36642b540678" containerName="registry-server"
Jul 16 11:59:27.692098 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:27.691963    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ffa5de0-7bd7-477e-a207-36642b540678" containerName="registry-server"
Jul 16 11:59:27.696227 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:27.696208    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"
Jul 16 11:59:27.700807 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:27.700783    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fwlpm\""
Jul 16 11:59:27.719495 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:27.719468    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"]
Jul 16 11:59:27.775314 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:27.775282    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfhc6\" (UniqueName: \"kubernetes.io/projected/bfaa4cb2-6902-449e-86be-bc983f29d812-kube-api-access-qfhc6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ddkv4\" (UID: \"bfaa4cb2-6902-449e-86be-bc983f29d812\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"
Jul 16 11:59:27.775449 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:27.775326    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4cb2-6902-449e-86be-bc983f29d812-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ddkv4\" (UID: \"bfaa4cb2-6902-449e-86be-bc983f29d812\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"
Jul 16 11:59:27.875662 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:27.875630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfhc6\" (UniqueName: \"kubernetes.io/projected/bfaa4cb2-6902-449e-86be-bc983f29d812-kube-api-access-qfhc6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ddkv4\" (UID: \"bfaa4cb2-6902-449e-86be-bc983f29d812\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"
Jul 16 11:59:27.875820 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:27.875674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4cb2-6902-449e-86be-bc983f29d812-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ddkv4\" (UID: \"bfaa4cb2-6902-449e-86be-bc983f29d812\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"
Jul 16 11:59:27.876123 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:27.876104    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4cb2-6902-449e-86be-bc983f29d812-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ddkv4\" (UID: \"bfaa4cb2-6902-449e-86be-bc983f29d812\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"
Jul 16 11:59:27.894624 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:27.894595    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfhc6\" (UniqueName: \"kubernetes.io/projected/bfaa4cb2-6902-449e-86be-bc983f29d812-kube-api-access-qfhc6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ddkv4\" (UID: \"bfaa4cb2-6902-449e-86be-bc983f29d812\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"
Jul 16 11:59:28.006286 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:28.006196    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"
Jul 16 11:59:28.166594 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:28.166562    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"]
Jul 16 11:59:28.169008 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:59:28.168981    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfaa4cb2_6902_449e_86be_bc983f29d812.slice/crio-d39e403fc5f6dc5ed89bb8e135f920a57a99a72388e882e916523a6b12cb9638 WatchSource:0}: Error finding container d39e403fc5f6dc5ed89bb8e135f920a57a99a72388e882e916523a6b12cb9638: Status 404 returned error can't find the container with id d39e403fc5f6dc5ed89bb8e135f920a57a99a72388e882e916523a6b12cb9638
Jul 16 11:59:28.280839 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:28.280744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4" event={"ID":"bfaa4cb2-6902-449e-86be-bc983f29d812","Type":"ContainerStarted","Data":"d39e403fc5f6dc5ed89bb8e135f920a57a99a72388e882e916523a6b12cb9638"}
Jul 16 11:59:30.049634 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:30.049597    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-9dczj"]
Jul 16 11:59:30.053436 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:30.053408    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9dczj"
Jul 16 11:59:30.056683 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:30.056650    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-s8bm9\""
Jul 16 11:59:30.076634 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:30.076604    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-9dczj"]
Jul 16 11:59:30.193757 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:30.193725    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf2s5\" (UniqueName: \"kubernetes.io/projected/cbaaae31-9490-4c9d-8ad7-21e84cf8c0db-kube-api-access-zf2s5\") pod \"authorino-operator-75cc8d5cb7-9dczj\" (UID: \"cbaaae31-9490-4c9d-8ad7-21e84cf8c0db\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-9dczj"
Jul 16 11:59:30.294397 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:30.294369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf2s5\" (UniqueName: \"kubernetes.io/projected/cbaaae31-9490-4c9d-8ad7-21e84cf8c0db-kube-api-access-zf2s5\") pod \"authorino-operator-75cc8d5cb7-9dczj\" (UID: \"cbaaae31-9490-4c9d-8ad7-21e84cf8c0db\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-9dczj"
Jul 16 11:59:30.307691 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:30.307631    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf2s5\" (UniqueName: \"kubernetes.io/projected/cbaaae31-9490-4c9d-8ad7-21e84cf8c0db-kube-api-access-zf2s5\") pod \"authorino-operator-75cc8d5cb7-9dczj\" (UID: \"cbaaae31-9490-4c9d-8ad7-21e84cf8c0db\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-9dczj"
Jul 16 11:59:30.367488 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:30.367458    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9dczj"
Jul 16 11:59:30.523164 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:30.523142    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-9dczj"]
Jul 16 11:59:30.524685 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:59:30.524648    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbaaae31_9490_4c9d_8ad7_21e84cf8c0db.slice/crio-0d95bf19ba38e13bcc76bed4edac7071b3e10ab4379789eb5de8f8058d3950eb WatchSource:0}: Error finding container 0d95bf19ba38e13bcc76bed4edac7071b3e10ab4379789eb5de8f8058d3950eb: Status 404 returned error can't find the container with id 0d95bf19ba38e13bcc76bed4edac7071b3e10ab4379789eb5de8f8058d3950eb
Jul 16 11:59:31.300488 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:31.300436    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9dczj" event={"ID":"cbaaae31-9490-4c9d-8ad7-21e84cf8c0db","Type":"ContainerStarted","Data":"0d95bf19ba38e13bcc76bed4edac7071b3e10ab4379789eb5de8f8058d3950eb"}
Jul 16 11:59:32.543662 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:32.543624    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9mmp"]
Jul 16 11:59:32.554258 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:32.554236    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9mmp"
Jul 16 11:59:32.558627 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:32.558602    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-gwtsk\""
Jul 16 11:59:32.558774 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:32.558672    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 16 11:59:32.565487 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:32.565467    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9mmp"]
Jul 16 11:59:32.719781 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:32.719748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swklz\" (UniqueName: \"kubernetes.io/projected/90410e5d-cd0c-4db8-9e2a-5f2cbe83d529-kube-api-access-swklz\") pod \"dns-operator-controller-manager-864dd78c4c-n9mmp\" (UID: \"90410e5d-cd0c-4db8-9e2a-5f2cbe83d529\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9mmp"
Jul 16 11:59:32.820294 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:32.820209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swklz\" (UniqueName: \"kubernetes.io/projected/90410e5d-cd0c-4db8-9e2a-5f2cbe83d529-kube-api-access-swklz\") pod \"dns-operator-controller-manager-864dd78c4c-n9mmp\" (UID: \"90410e5d-cd0c-4db8-9e2a-5f2cbe83d529\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9mmp"
Jul 16 11:59:32.838618 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:32.838577    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swklz\" (UniqueName: \"kubernetes.io/projected/90410e5d-cd0c-4db8-9e2a-5f2cbe83d529-kube-api-access-swklz\") pod \"dns-operator-controller-manager-864dd78c4c-n9mmp\" (UID: \"90410e5d-cd0c-4db8-9e2a-5f2cbe83d529\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9mmp"
Jul 16 11:59:32.866870 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:32.866824    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9mmp"
Jul 16 11:59:34.278084 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:34.278050    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9mmp"]
Jul 16 11:59:34.281439 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:59:34.281408    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90410e5d_cd0c_4db8_9e2a_5f2cbe83d529.slice/crio-0d71d78c077e45b5f1ec974e9298558519adca9a7320b7ba20a41ad4a0d0e2d0 WatchSource:0}: Error finding container 0d71d78c077e45b5f1ec974e9298558519adca9a7320b7ba20a41ad4a0d0e2d0: Status 404 returned error can't find the container with id 0d71d78c077e45b5f1ec974e9298558519adca9a7320b7ba20a41ad4a0d0e2d0
Jul 16 11:59:34.312852 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:34.312825    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9mmp" event={"ID":"90410e5d-cd0c-4db8-9e2a-5f2cbe83d529","Type":"ContainerStarted","Data":"0d71d78c077e45b5f1ec974e9298558519adca9a7320b7ba20a41ad4a0d0e2d0"}
Jul 16 11:59:34.314288 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:34.314267    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9dczj" event={"ID":"cbaaae31-9490-4c9d-8ad7-21e84cf8c0db","Type":"ContainerStarted","Data":"8692ed94691f1b69a39a50e92d435d22a2a70cc62f104ea9cbedf335c728c4f4"}
Jul 16 11:59:34.314414 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:34.314393    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9dczj"
Jul 16 11:59:34.315894 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:34.315870    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4" event={"ID":"bfaa4cb2-6902-449e-86be-bc983f29d812","Type":"ContainerStarted","Data":"3b3d69b4f82799a29868ac66ebc68e87ddc613187b73c0267f72542c566e79d3"}
Jul 16 11:59:34.316005 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:34.315994    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"
Jul 16 11:59:34.342767 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:34.342726    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9dczj" podStartSLOduration=0.739499505 podStartE2EDuration="4.342713311s" totalImagesPullingTime="3.603213806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:59:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:59:34.340752816 +0000 UTC m=+515.641529913" watchObservedRunningTime="2026-07-16 11:59:34.342713311 +0000 UTC m=+515.643490409"
Jul 16 11:59:34.370363 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:34.370321    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4" podStartSLOduration=1.397731192 podStartE2EDuration="7.370307766s" totalImagesPullingTime="5.972576574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:59:34.369935804 +0000 UTC m=+515.670712901" watchObservedRunningTime="2026-07-16 11:59:34.370307766 +0000 UTC m=+515.671084863"
Jul 16 11:59:35.876488 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:35.874266    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd"]
Jul 16 11:59:35.878974 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:35.878942    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd"
Jul 16 11:59:35.882471 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:35.882448    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-shmll\""
Jul 16 11:59:35.899726 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:35.899699    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd"]
Jul 16 11:59:36.049886 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:36.049846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h95rx\" (UniqueName: \"kubernetes.io/projected/f4571956-e4fa-4dda-a7ee-06f28086f073-kube-api-access-h95rx\") pod \"limitador-operator-controller-manager-859d7b78bd-8thzd\" (UID: \"f4571956-e4fa-4dda-a7ee-06f28086f073\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd"
Jul 16 11:59:36.151055 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:36.150920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h95rx\" (UniqueName: \"kubernetes.io/projected/f4571956-e4fa-4dda-a7ee-06f28086f073-kube-api-access-h95rx\") pod \"limitador-operator-controller-manager-859d7b78bd-8thzd\" (UID: \"f4571956-e4fa-4dda-a7ee-06f28086f073\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd"
Jul 16 11:59:36.160703 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:36.160662    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h95rx\" (UniqueName: \"kubernetes.io/projected/f4571956-e4fa-4dda-a7ee-06f28086f073-kube-api-access-h95rx\") pod \"limitador-operator-controller-manager-859d7b78bd-8thzd\" (UID: \"f4571956-e4fa-4dda-a7ee-06f28086f073\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd"
Jul 16 11:59:36.190912 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:36.190875    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd"
Jul 16 11:59:36.329977 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:36.329943    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9mmp" event={"ID":"90410e5d-cd0c-4db8-9e2a-5f2cbe83d529","Type":"ContainerStarted","Data":"78976b3304430ae0a428a5879427ffbd23acab22c09fbc8fd86b1a4568251b48"}
Jul 16 11:59:36.330138 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:36.330054    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9mmp"
Jul 16 11:59:36.361958 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:59:36.361932    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4571956_e4fa_4dda_a7ee_06f28086f073.slice/crio-dc1a37c1907a0ed940ef52bfcf4ba643ecdd87eff48abbea9509047be001d861 WatchSource:0}: Error finding container dc1a37c1907a0ed940ef52bfcf4ba643ecdd87eff48abbea9509047be001d861: Status 404 returned error can't find the container with id dc1a37c1907a0ed940ef52bfcf4ba643ecdd87eff48abbea9509047be001d861
Jul 16 11:59:36.362134 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:36.362114    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd"]
Jul 16 11:59:37.277023 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:37.276958    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9mmp" podStartSLOduration=3.339981028 podStartE2EDuration="5.276935437s" totalImagesPullingTime="1.936954409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:59:36.369555477 +0000 UTC m=+517.670332574" watchObservedRunningTime="2026-07-16 11:59:37.276935437 +0000 UTC m=+518.577712536"
Jul 16 11:59:37.278792 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:37.278765    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-s7rhm"]
Jul 16 11:59:37.282573 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:37.282554    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rhm"
Jul 16 11:59:37.284938 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:37.284909    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 16 11:59:37.285417 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:37.285394    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 16 11:59:37.285558 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:37.285454    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xm2kv\""
Jul 16 11:59:37.301158 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:37.300927    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-s7rhm"]
Jul 16 11:59:37.336459 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:37.336418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd" event={"ID":"f4571956-e4fa-4dda-a7ee-06f28086f073","Type":"ContainerStarted","Data":"dc1a37c1907a0ed940ef52bfcf4ba643ecdd87eff48abbea9509047be001d861"}
Jul 16 11:59:37.360282 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:37.360245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2brg\" (UniqueName: \"kubernetes.io/projected/73e603da-fca7-4e9f-82a2-25b0c7b3076c-kube-api-access-b2brg\") pod \"kuadrant-console-plugin-66b5774985-s7rhm\" (UID: \"73e603da-fca7-4e9f-82a2-25b0c7b3076c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rhm"
Jul 16 11:59:37.360408 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:37.360304    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/73e603da-fca7-4e9f-82a2-25b0c7b3076c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-s7rhm\" (UID: \"73e603da-fca7-4e9f-82a2-25b0c7b3076c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rhm"
Jul 16 11:59:37.360408 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:37.360335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/73e603da-fca7-4e9f-82a2-25b0c7b3076c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-s7rhm\" (UID: \"73e603da-fca7-4e9f-82a2-25b0c7b3076c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rhm"
Jul 16 11:59:37.461321 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:37.461292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/73e603da-fca7-4e9f-82a2-25b0c7b3076c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-s7rhm\" (UID: \"73e603da-fca7-4e9f-82a2-25b0c7b3076c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rhm"
Jul 16 11:59:37.461321 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:37.461323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/73e603da-fca7-4e9f-82a2-25b0c7b3076c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-s7rhm\" (UID: \"73e603da-fca7-4e9f-82a2-25b0c7b3076c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rhm"
Jul 16 11:59:37.461511 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:37.461396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2brg\" (UniqueName: \"kubernetes.io/projected/73e603da-fca7-4e9f-82a2-25b0c7b3076c-kube-api-access-b2brg\") pod \"kuadrant-console-plugin-66b5774985-s7rhm\" (UID: \"73e603da-fca7-4e9f-82a2-25b0c7b3076c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rhm"
Jul 16 11:59:37.462006 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:37.461988    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/73e603da-fca7-4e9f-82a2-25b0c7b3076c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-s7rhm\" (UID: \"73e603da-fca7-4e9f-82a2-25b0c7b3076c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rhm"
Jul 16 11:59:37.463766 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:37.463737    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/73e603da-fca7-4e9f-82a2-25b0c7b3076c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-s7rhm\" (UID: \"73e603da-fca7-4e9f-82a2-25b0c7b3076c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rhm"
Jul 16 11:59:37.477367 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:37.477343    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2brg\" (UniqueName: \"kubernetes.io/projected/73e603da-fca7-4e9f-82a2-25b0c7b3076c-kube-api-access-b2brg\") pod \"kuadrant-console-plugin-66b5774985-s7rhm\" (UID: \"73e603da-fca7-4e9f-82a2-25b0c7b3076c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rhm"
Jul 16 11:59:37.593865 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:37.593825    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rhm"
Jul 16 11:59:37.986224 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:59:37.986193    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73e603da_fca7_4e9f_82a2_25b0c7b3076c.slice/crio-1dfff2b7cfd63df1c24088e8481905e8909dfcca81cc2a96feb698f0aa51920e WatchSource:0}: Error finding container 1dfff2b7cfd63df1c24088e8481905e8909dfcca81cc2a96feb698f0aa51920e: Status 404 returned error can't find the container with id 1dfff2b7cfd63df1c24088e8481905e8909dfcca81cc2a96feb698f0aa51920e
Jul 16 11:59:37.986622 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:37.986601    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-s7rhm"]
Jul 16 11:59:38.342428 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:38.342397    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd" event={"ID":"f4571956-e4fa-4dda-a7ee-06f28086f073","Type":"ContainerStarted","Data":"70940e538bb91fc3763cead609d9a1ba3c8d5bd3c79f32be09b220865f212425"}
Jul 16 11:59:38.342831 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:38.342549    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd"
Jul 16 11:59:38.343647 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:38.343629    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rhm" event={"ID":"73e603da-fca7-4e9f-82a2-25b0c7b3076c","Type":"ContainerStarted","Data":"1dfff2b7cfd63df1c24088e8481905e8909dfcca81cc2a96feb698f0aa51920e"}
Jul 16 11:59:38.395447 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:38.395403    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd" podStartSLOduration=1.861438245 podStartE2EDuration="3.395387372s" totalImagesPullingTime="1.533949127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:59:38.394955175 +0000 UTC m=+519.695732272" watchObservedRunningTime="2026-07-16 11:59:38.395387372 +0000 UTC m=+519.696164469"
Jul 16 11:59:45.324758 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:45.324725    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9dczj"
Jul 16 11:59:45.325263 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:45.324879    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"
Jul 16 11:59:47.339556 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:47.339527    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9mmp"
Jul 16 11:59:48.077251 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.077213    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"]
Jul 16 11:59:48.077503 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.077475    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4" podUID="bfaa4cb2-6902-449e-86be-bc983f29d812" containerName="manager" containerID="cri-o://3b3d69b4f82799a29868ac66ebc68e87ddc613187b73c0267f72542c566e79d3" gracePeriod=2
Jul 16 11:59:48.096127 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.096087    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"]
Jul 16 11:59:48.181298 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.181267    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd"]
Jul 16 11:59:48.181687 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.181518    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd" podUID="f4571956-e4fa-4dda-a7ee-06f28086f073" containerName="manager" containerID="cri-o://70940e538bb91fc3763cead609d9a1ba3c8d5bd3c79f32be09b220865f212425" gracePeriod=2
Jul 16 11:59:48.183249 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.183224    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd"
Jul 16 11:59:48.210542 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:59:48.210502    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ddkv4\" is forbidden: User \"system:node:ip-10-0-129-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-163.ec2.internal' and this object" podUID="bfaa4cb2-6902-449e-86be-bc983f29d812" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"
Jul 16 11:59:48.212748 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.212723    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd"]
Jul 16 11:59:48.219802 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:59:48.219774    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ddkv4\" is forbidden: User \"system:node:ip-10-0-129-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-163.ec2.internal' and this object" podUID="bfaa4cb2-6902-449e-86be-bc983f29d812" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"
Jul 16 11:59:48.222031 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:59:48.222001    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8thzd\" is forbidden: User \"system:node:ip-10-0-129-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-163.ec2.internal' and this object" podUID="f4571956-e4fa-4dda-a7ee-06f28086f073" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd"
Jul 16 11:59:48.248401 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.248382    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6"]
Jul 16 11:59:48.248702 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.248689    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfaa4cb2-6902-449e-86be-bc983f29d812" containerName="manager"
Jul 16 11:59:48.248702 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.248703    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfaa4cb2-6902-449e-86be-bc983f29d812" containerName="manager"
Jul 16 11:59:48.248819 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.248721    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4571956-e4fa-4dda-a7ee-06f28086f073" containerName="manager"
Jul 16 11:59:48.248819 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.248727    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4571956-e4fa-4dda-a7ee-06f28086f073" containerName="manager"
Jul 16 11:59:48.248819 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.248781    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4571956-e4fa-4dda-a7ee-06f28086f073" containerName="manager"
Jul 16 11:59:48.248819 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.248789    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfaa4cb2-6902-449e-86be-bc983f29d812" containerName="manager"
Jul 16 11:59:48.255055 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.255020    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6"
Jul 16 11:59:48.257217 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:59:48.257180    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ddkv4\" is forbidden: User \"system:node:ip-10-0-129-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-163.ec2.internal' and this object" podUID="bfaa4cb2-6902-449e-86be-bc983f29d812" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"
Jul 16 11:59:48.264834 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:59:48.264802    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8thzd\" is forbidden: User \"system:node:ip-10-0-129-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-163.ec2.internal' and this object" podUID="f4571956-e4fa-4dda-a7ee-06f28086f073" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd"
Jul 16 11:59:48.314454 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.314427    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72mxq"]
Jul 16 11:59:48.321170 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.321108    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6"]
Jul 16 11:59:48.321304 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.321224    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72mxq"
Jul 16 11:59:48.355430 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.355396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsvjg\" (UniqueName: \"kubernetes.io/projected/3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def-kube-api-access-jsvjg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-x57g6\" (UID: \"3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6"
Jul 16 11:59:48.355770 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.355468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-x57g6\" (UID: \"3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6"
Jul 16 11:59:48.355770 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.355528    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kntd\" (UniqueName: \"kubernetes.io/projected/ab96b12a-1d9b-41fa-b16c-4193c471338d-kube-api-access-8kntd\") pod \"limitador-operator-controller-manager-859d7b78bd-72mxq\" (UID: \"ab96b12a-1d9b-41fa-b16c-4193c471338d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72mxq"
Jul 16 11:59:48.364014 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.363992    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72mxq"]
Jul 16 11:59:48.370001 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:59:48.369969    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ddkv4\" is forbidden: User \"system:node:ip-10-0-129-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-163.ec2.internal' and this object" podUID="bfaa4cb2-6902-449e-86be-bc983f29d812" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"
Jul 16 11:59:48.375926 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:59:48.375891    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8thzd\" is forbidden: User \"system:node:ip-10-0-129-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-163.ec2.internal' and this object" podUID="f4571956-e4fa-4dda-a7ee-06f28086f073" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd"
Jul 16 11:59:48.456995 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.456959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsvjg\" (UniqueName: \"kubernetes.io/projected/3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def-kube-api-access-jsvjg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-x57g6\" (UID: \"3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6"
Jul 16 11:59:48.457201 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.457022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-x57g6\" (UID: \"3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6"
Jul 16 11:59:48.457201 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.457102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kntd\" (UniqueName: \"kubernetes.io/projected/ab96b12a-1d9b-41fa-b16c-4193c471338d-kube-api-access-8kntd\") pod \"limitador-operator-controller-manager-859d7b78bd-72mxq\" (UID: \"ab96b12a-1d9b-41fa-b16c-4193c471338d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72mxq"
Jul 16 11:59:48.457454 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.457425    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-x57g6\" (UID: \"3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6"
Jul 16 11:59:48.485499 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.485464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsvjg\" (UniqueName: \"kubernetes.io/projected/3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def-kube-api-access-jsvjg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-x57g6\" (UID: \"3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6"
Jul 16 11:59:48.488735 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.488712    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kntd\" (UniqueName: \"kubernetes.io/projected/ab96b12a-1d9b-41fa-b16c-4193c471338d-kube-api-access-8kntd\") pod \"limitador-operator-controller-manager-859d7b78bd-72mxq\" (UID: \"ab96b12a-1d9b-41fa-b16c-4193c471338d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72mxq"
Jul 16 11:59:48.567740 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.567707    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6"
Jul 16 11:59:48.632590 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:48.632558    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72mxq"
Jul 16 11:59:49.301802 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:59:49.301766    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ddkv4\" is forbidden: User \"system:node:ip-10-0-129-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-163.ec2.internal' and this object" podUID="bfaa4cb2-6902-449e-86be-bc983f29d812" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"
Jul 16 11:59:49.304601 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:59:49.304568    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8thzd\" is forbidden: User \"system:node:ip-10-0-129-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-163.ec2.internal' and this object" podUID="f4571956-e4fa-4dda-a7ee-06f28086f073" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd"
Jul 16 11:59:51.475591 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:51.475462    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p"]
Jul 16 11:59:51.510391 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:51.510354    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p"]
Jul 16 11:59:51.510583 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:51.510491    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p"
Jul 16 11:59:51.585852 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:51.585820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz26j\" (UniqueName: \"kubernetes.io/projected/160effe3-5586-4035-bde2-a3ae65043966-kube-api-access-vz26j\") pod \"kuadrant-operator-controller-manager-86878c769f-6nf2p\" (UID: \"160effe3-5586-4035-bde2-a3ae65043966\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p"
Jul 16 11:59:51.586018 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:51.585891    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/160effe3-5586-4035-bde2-a3ae65043966-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6nf2p\" (UID: \"160effe3-5586-4035-bde2-a3ae65043966\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p"
Jul 16 11:59:51.687359 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:51.687320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/160effe3-5586-4035-bde2-a3ae65043966-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6nf2p\" (UID: \"160effe3-5586-4035-bde2-a3ae65043966\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p"
Jul 16 11:59:51.687549 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:51.687418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz26j\" (UniqueName: \"kubernetes.io/projected/160effe3-5586-4035-bde2-a3ae65043966-kube-api-access-vz26j\") pod \"kuadrant-operator-controller-manager-86878c769f-6nf2p\" (UID: \"160effe3-5586-4035-bde2-a3ae65043966\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p"
Jul 16 11:59:51.687793 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:51.687763    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/160effe3-5586-4035-bde2-a3ae65043966-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6nf2p\" (UID: \"160effe3-5586-4035-bde2-a3ae65043966\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p"
Jul 16 11:59:51.706959 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:51.706928    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz26j\" (UniqueName: \"kubernetes.io/projected/160effe3-5586-4035-bde2-a3ae65043966-kube-api-access-vz26j\") pod \"kuadrant-operator-controller-manager-86878c769f-6nf2p\" (UID: \"160effe3-5586-4035-bde2-a3ae65043966\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p"
Jul 16 11:59:51.823342 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:51.823307    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p"
Jul 16 11:59:58.829926 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:58.829892    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd"
Jul 16 11:59:58.841604 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:58.841493    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"
Jul 16 11:59:58.850552 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:58.850422    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72mxq"]
Jul 16 11:59:58.853409 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:59:58.853374    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab96b12a_1d9b_41fa_b16c_4193c471338d.slice/crio-9ba848aa7f250853b2cca9176ce958e88c30ab8dfe2e46b7ba2c9520b6cf8001 WatchSource:0}: Error finding container 9ba848aa7f250853b2cca9176ce958e88c30ab8dfe2e46b7ba2c9520b6cf8001: Status 404 returned error can't find the container with id 9ba848aa7f250853b2cca9176ce958e88c30ab8dfe2e46b7ba2c9520b6cf8001
Jul 16 11:59:58.953451 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:58.953429    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h95rx\" (UniqueName: \"kubernetes.io/projected/f4571956-e4fa-4dda-a7ee-06f28086f073-kube-api-access-h95rx\") pod \"f4571956-e4fa-4dda-a7ee-06f28086f073\" (UID: \"f4571956-e4fa-4dda-a7ee-06f28086f073\") "
Jul 16 11:59:58.953562 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:58.953473    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfhc6\" (UniqueName: \"kubernetes.io/projected/bfaa4cb2-6902-449e-86be-bc983f29d812-kube-api-access-qfhc6\") pod \"bfaa4cb2-6902-449e-86be-bc983f29d812\" (UID: \"bfaa4cb2-6902-449e-86be-bc983f29d812\") "
Jul 16 11:59:58.953616 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:58.953563    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4cb2-6902-449e-86be-bc983f29d812-extensions-socket-volume\") pod \"bfaa4cb2-6902-449e-86be-bc983f29d812\" (UID: \"bfaa4cb2-6902-449e-86be-bc983f29d812\") "
Jul 16 11:59:58.954116 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:58.954082    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaa4cb2-6902-449e-86be-bc983f29d812-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bfaa4cb2-6902-449e-86be-bc983f29d812" (UID: "bfaa4cb2-6902-449e-86be-bc983f29d812"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 11:59:58.955406 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:58.955384    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4571956-e4fa-4dda-a7ee-06f28086f073-kube-api-access-h95rx" (OuterVolumeSpecName: "kube-api-access-h95rx") pod "f4571956-e4fa-4dda-a7ee-06f28086f073" (UID: "f4571956-e4fa-4dda-a7ee-06f28086f073"). InnerVolumeSpecName "kube-api-access-h95rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 11:59:58.955479 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:58.955402    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfaa4cb2-6902-449e-86be-bc983f29d812-kube-api-access-qfhc6" (OuterVolumeSpecName: "kube-api-access-qfhc6") pod "bfaa4cb2-6902-449e-86be-bc983f29d812" (UID: "bfaa4cb2-6902-449e-86be-bc983f29d812"). InnerVolumeSpecName "kube-api-access-qfhc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 11:59:59.054404 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.054375    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h95rx\" (UniqueName: \"kubernetes.io/projected/f4571956-e4fa-4dda-a7ee-06f28086f073-kube-api-access-h95rx\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 11:59:59.054404 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.054404    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfhc6\" (UniqueName: \"kubernetes.io/projected/bfaa4cb2-6902-449e-86be-bc983f29d812-kube-api-access-qfhc6\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 11:59:59.054575 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.054417    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4cb2-6902-449e-86be-bc983f29d812-extensions-socket-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 11:59:59.083590 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.083424    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p"]
Jul 16 11:59:59.085126 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.085105    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6"]
Jul 16 11:59:59.086223 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:59:59.086194    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5aa9dc_4f0d_456d_bb39_bf54d8ae8def.slice/crio-adf09e4b9cf3eaaf9b148b1e7fcec7171a7f7555930a73ef71105dd740f1a1a4 WatchSource:0}: Error finding container adf09e4b9cf3eaaf9b148b1e7fcec7171a7f7555930a73ef71105dd740f1a1a4: Status 404 returned error can't find the container with id adf09e4b9cf3eaaf9b148b1e7fcec7171a7f7555930a73ef71105dd740f1a1a4
Jul 16 11:59:59.086752 ip-10-0-129-163 kubenswrapper[2578]: W0716 11:59:59.086733    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160effe3_5586_4035_bde2_a3ae65043966.slice/crio-bf734c915ea5c3eedb0cd1f3825be32b82cf9522b34494cdfd0532a4a7c1b6d9 WatchSource:0}: Error finding container bf734c915ea5c3eedb0cd1f3825be32b82cf9522b34494cdfd0532a4a7c1b6d9: Status 404 returned error can't find the container with id bf734c915ea5c3eedb0cd1f3825be32b82cf9522b34494cdfd0532a4a7c1b6d9
Jul 16 11:59:59.109158 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.109137    2578 scope.go:117] "RemoveContainer" containerID="1be4d7ce5b50bf7ede15ab54350199cc4ade99a8e0841d4454b851f954cf2332"
Jul 16 11:59:59.299896 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.299864    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfaa4cb2-6902-449e-86be-bc983f29d812" path="/var/lib/kubelet/pods/bfaa4cb2-6902-449e-86be-bc983f29d812/volumes"
Jul 16 11:59:59.300305 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.300287    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4571956-e4fa-4dda-a7ee-06f28086f073" path="/var/lib/kubelet/pods/f4571956-e4fa-4dda-a7ee-06f28086f073/volumes"
Jul 16 11:59:59.453206 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.453122    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72mxq" event={"ID":"ab96b12a-1d9b-41fa-b16c-4193c471338d","Type":"ContainerStarted","Data":"64bf0130a0cbc2b4c647a86ffa6e712d7904fab878b9d583d5f31de609e5a462"}
Jul 16 11:59:59.453206 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.453173    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72mxq" event={"ID":"ab96b12a-1d9b-41fa-b16c-4193c471338d","Type":"ContainerStarted","Data":"9ba848aa7f250853b2cca9176ce958e88c30ab8dfe2e46b7ba2c9520b6cf8001"}
Jul 16 11:59:59.453409 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.453217    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72mxq"
Jul 16 11:59:59.454474 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.454451    2578 generic.go:356] "Generic (PLEG): container finished" podID="bfaa4cb2-6902-449e-86be-bc983f29d812" containerID="3b3d69b4f82799a29868ac66ebc68e87ddc613187b73c0267f72542c566e79d3" exitCode=0
Jul 16 11:59:59.454598 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.454492    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ddkv4"
Jul 16 11:59:59.454598 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.454536    2578 scope.go:117] "RemoveContainer" containerID="3b3d69b4f82799a29868ac66ebc68e87ddc613187b73c0267f72542c566e79d3"
Jul 16 11:59:59.456465 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.456433    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p" event={"ID":"160effe3-5586-4035-bde2-a3ae65043966","Type":"ContainerStarted","Data":"fe9953a0befc2e9c20fc15f3ac4fbad6c27f52e6863574b40c428ec4c3785190"}
Jul 16 11:59:59.456635 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.456487    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p" event={"ID":"160effe3-5586-4035-bde2-a3ae65043966","Type":"ContainerStarted","Data":"bf734c915ea5c3eedb0cd1f3825be32b82cf9522b34494cdfd0532a4a7c1b6d9"}
Jul 16 11:59:59.456635 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.456545    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p"
Jul 16 11:59:59.458476 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.458455    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6" event={"ID":"3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def","Type":"ContainerStarted","Data":"337a17f35dbf568155561b1a430c5c12ab94be3da059f92e81eb2597475cda85"}
Jul 16 11:59:59.458571 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.458503    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6"
Jul 16 11:59:59.458571 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.458527    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6" event={"ID":"3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def","Type":"ContainerStarted","Data":"adf09e4b9cf3eaaf9b148b1e7fcec7171a7f7555930a73ef71105dd740f1a1a4"}
Jul 16 11:59:59.459813 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.459792    2578 generic.go:356] "Generic (PLEG): container finished" podID="f4571956-e4fa-4dda-a7ee-06f28086f073" containerID="70940e538bb91fc3763cead609d9a1ba3c8d5bd3c79f32be09b220865f212425" exitCode=0
Jul 16 11:59:59.459920 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.459826    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8thzd"
Jul 16 11:59:59.461380 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.461360    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rhm" event={"ID":"73e603da-fca7-4e9f-82a2-25b0c7b3076c","Type":"ContainerStarted","Data":"a1732f4a397b4944de0e3c6fd17a9e78e708247aba24bac44c24ee03d96ac660"}
Jul 16 11:59:59.463968 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.463953    2578 scope.go:117] "RemoveContainer" containerID="3b3d69b4f82799a29868ac66ebc68e87ddc613187b73c0267f72542c566e79d3"
Jul 16 11:59:59.464207 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:59:59.464189    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b3d69b4f82799a29868ac66ebc68e87ddc613187b73c0267f72542c566e79d3\": container with ID starting with 3b3d69b4f82799a29868ac66ebc68e87ddc613187b73c0267f72542c566e79d3 not found: ID does not exist" containerID="3b3d69b4f82799a29868ac66ebc68e87ddc613187b73c0267f72542c566e79d3"
Jul 16 11:59:59.464283 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.464220    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b3d69b4f82799a29868ac66ebc68e87ddc613187b73c0267f72542c566e79d3"} err="failed to get container status \"3b3d69b4f82799a29868ac66ebc68e87ddc613187b73c0267f72542c566e79d3\": rpc error: code = NotFound desc = could not find container \"3b3d69b4f82799a29868ac66ebc68e87ddc613187b73c0267f72542c566e79d3\": container with ID starting with 3b3d69b4f82799a29868ac66ebc68e87ddc613187b73c0267f72542c566e79d3 not found: ID does not exist"
Jul 16 11:59:59.464283 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.464242    2578 scope.go:117] "RemoveContainer" containerID="70940e538bb91fc3763cead609d9a1ba3c8d5bd3c79f32be09b220865f212425"
Jul 16 11:59:59.471302 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.471288    2578 scope.go:117] "RemoveContainer" containerID="70940e538bb91fc3763cead609d9a1ba3c8d5bd3c79f32be09b220865f212425"
Jul 16 11:59:59.471510 ip-10-0-129-163 kubenswrapper[2578]: E0716 11:59:59.471495    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70940e538bb91fc3763cead609d9a1ba3c8d5bd3c79f32be09b220865f212425\": container with ID starting with 70940e538bb91fc3763cead609d9a1ba3c8d5bd3c79f32be09b220865f212425 not found: ID does not exist" containerID="70940e538bb91fc3763cead609d9a1ba3c8d5bd3c79f32be09b220865f212425"
Jul 16 11:59:59.471562 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.471515    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70940e538bb91fc3763cead609d9a1ba3c8d5bd3c79f32be09b220865f212425"} err="failed to get container status \"70940e538bb91fc3763cead609d9a1ba3c8d5bd3c79f32be09b220865f212425\": rpc error: code = NotFound desc = could not find container \"70940e538bb91fc3763cead609d9a1ba3c8d5bd3c79f32be09b220865f212425\": container with ID starting with 70940e538bb91fc3763cead609d9a1ba3c8d5bd3c79f32be09b220865f212425 not found: ID does not exist"
Jul 16 11:59:59.494527 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.494479    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72mxq" podStartSLOduration=11.494465195 podStartE2EDuration="11.494465195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:59:59.491738416 +0000 UTC m=+540.792515516" watchObservedRunningTime="2026-07-16 11:59:59.494465195 +0000 UTC m=+540.795242365"
Jul 16 11:59:59.539239 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.539183    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6" podStartSLOduration=11.539164555 podStartE2EDuration="11.539164555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:59:59.538204831 +0000 UTC m=+540.838981947" watchObservedRunningTime="2026-07-16 11:59:59.539164555 +0000 UTC m=+540.839941653"
Jul 16 11:59:59.697089 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.697015    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rhm" podStartSLOduration=1.950347711 podStartE2EDuration="22.696998213s" totalImagesPullingTime="20.746650502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:59:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:59:59.695384895 +0000 UTC m=+540.996161995" watchObservedRunningTime="2026-07-16 11:59:59.696998213 +0000 UTC m=+540.997775319"
Jul 16 11:59:59.701055 ip-10-0-129-163 kubenswrapper[2578]: I0716 11:59:59.697357    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p" podStartSLOduration=8.697348615 podStartE2EDuration="8.697348615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:59:59.624186705 +0000 UTC m=+540.924963823" watchObservedRunningTime="2026-07-16 11:59:59.697348615 +0000 UTC m=+540.998125713"
Jul 16 12:00:10.471176 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:10.471143    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6"
Jul 16 12:00:10.471631 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:10.471222    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p"
Jul 16 12:00:10.471631 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:10.471396    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72mxq"
Jul 16 12:00:10.683993 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:10.683962    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6"]
Jul 16 12:00:10.684185 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:10.684164    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6" podUID="3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def" containerName="manager" containerID="cri-o://337a17f35dbf568155561b1a430c5c12ab94be3da059f92e81eb2597475cda85" gracePeriod=10
Jul 16 12:00:10.934009 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:10.933986    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6"
Jul 16 12:00:11.047279 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:11.047251    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def-extensions-socket-volume\") pod \"3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def\" (UID: \"3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def\") "
Jul 16 12:00:11.047421 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:11.047290    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsvjg\" (UniqueName: \"kubernetes.io/projected/3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def-kube-api-access-jsvjg\") pod \"3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def\" (UID: \"3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def\") "
Jul 16 12:00:11.047660 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:11.047637    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def" (UID: "3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:00:11.049364 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:11.049342    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def-kube-api-access-jsvjg" (OuterVolumeSpecName: "kube-api-access-jsvjg") pod "3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def" (UID: "3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def"). InnerVolumeSpecName "kube-api-access-jsvjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:00:11.147776 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:11.147752    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def-extensions-socket-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:00:11.147776 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:11.147773    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsvjg\" (UniqueName: \"kubernetes.io/projected/3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def-kube-api-access-jsvjg\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:00:11.518608 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:11.518576    2578 generic.go:356] "Generic (PLEG): container finished" podID="3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def" containerID="337a17f35dbf568155561b1a430c5c12ab94be3da059f92e81eb2597475cda85" exitCode=0
Jul 16 12:00:11.519008 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:11.518619    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6"
Jul 16 12:00:11.519008 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:11.518664    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6" event={"ID":"3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def","Type":"ContainerDied","Data":"337a17f35dbf568155561b1a430c5c12ab94be3da059f92e81eb2597475cda85"}
Jul 16 12:00:11.519008 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:11.518711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6" event={"ID":"3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def","Type":"ContainerDied","Data":"adf09e4b9cf3eaaf9b148b1e7fcec7171a7f7555930a73ef71105dd740f1a1a4"}
Jul 16 12:00:11.519008 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:11.518734    2578 scope.go:117] "RemoveContainer" containerID="337a17f35dbf568155561b1a430c5c12ab94be3da059f92e81eb2597475cda85"
Jul 16 12:00:11.526754 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:11.526736    2578 scope.go:117] "RemoveContainer" containerID="337a17f35dbf568155561b1a430c5c12ab94be3da059f92e81eb2597475cda85"
Jul 16 12:00:11.527026 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:00:11.527009    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"337a17f35dbf568155561b1a430c5c12ab94be3da059f92e81eb2597475cda85\": container with ID starting with 337a17f35dbf568155561b1a430c5c12ab94be3da059f92e81eb2597475cda85 not found: ID does not exist" containerID="337a17f35dbf568155561b1a430c5c12ab94be3da059f92e81eb2597475cda85"
Jul 16 12:00:11.527091 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:11.527052    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337a17f35dbf568155561b1a430c5c12ab94be3da059f92e81eb2597475cda85"} err="failed to get container status \"337a17f35dbf568155561b1a430c5c12ab94be3da059f92e81eb2597475cda85\": rpc error: code = NotFound desc = could not find container \"337a17f35dbf568155561b1a430c5c12ab94be3da059f92e81eb2597475cda85\": container with ID starting with 337a17f35dbf568155561b1a430c5c12ab94be3da059f92e81eb2597475cda85 not found: ID does not exist"
Jul 16 12:00:11.582121 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:11.582018    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6"]
Jul 16 12:00:11.586506 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:11.586473    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x57g6"]
Jul 16 12:00:13.299410 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:13.299378    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def" path="/var/lib/kubelet/pods/3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def/volumes"
Jul 16 12:00:27.355991 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.355961    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"]
Jul 16 12:00:27.356434 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.356305    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def" containerName="manager"
Jul 16 12:00:27.356434 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.356316    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def" containerName="manager"
Jul 16 12:00:27.356434 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.356369    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c5aa9dc-4f0d-456d-bb39-bf54d8ae8def" containerName="manager"
Jul 16 12:00:27.516490 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.516458    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"]
Jul 16 12:00:27.516659 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.516558    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.519525 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.519503    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pv7f6\""
Jul 16 12:00:27.689717 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.689639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.689717 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.689684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.689717 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.689712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.689928 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.689778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.689928 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.689819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.689928 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.689836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkw9r\" (UniqueName: \"kubernetes.io/projected/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-kube-api-access-gkw9r\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.689928 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.689875    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.689928 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.689898    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.690096 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.689931    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.790591 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.790551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.790773 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.790605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.790773 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.790638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.790773 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.790671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.790773 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.790698    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.790988 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.790832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkw9r\" (UniqueName: \"kubernetes.io/projected/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-kube-api-access-gkw9r\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.790988 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.790891    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.790988 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.790921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.790988 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.790952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.791234 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.791095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.791284 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.791238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.791367 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.791350    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.791452 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.791423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.791557 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.791534    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.793109 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.793075    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.793276 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.793249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.801123 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.801097    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.801930 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.801909    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkw9r\" (UniqueName: \"kubernetes.io/projected/2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421-kube-api-access-gkw9r\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bzksk\" (UID: \"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:27.826883 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:27.826862    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:28.007167 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:28.007133    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"]
Jul 16 12:00:28.010999 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:00:28.010967    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf9ca58_d9ac_4b47_9aa3_2a177fbb0421.slice/crio-eb641fe27453497a5fdbd22329a2af0304fb9bd27e6214e0fa97405d192c0c07 WatchSource:0}: Error finding container eb641fe27453497a5fdbd22329a2af0304fb9bd27e6214e0fa97405d192c0c07: Status 404 returned error can't find the container with id eb641fe27453497a5fdbd22329a2af0304fb9bd27e6214e0fa97405d192c0c07
Jul 16 12:00:28.013189 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:28.013139    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:00:28.013294 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:28.013246    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:00:28.013294 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:28.013277    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:00:28.595849 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:28.595809    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk" event={"ID":"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421","Type":"ContainerStarted","Data":"b4aa654b234f0db424113ab4f111ac9f674f452d607a494a0ef15ddbc7600fbb"}
Jul 16 12:00:28.596285 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:28.595879    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk" event={"ID":"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421","Type":"ContainerStarted","Data":"eb641fe27453497a5fdbd22329a2af0304fb9bd27e6214e0fa97405d192c0c07"}
Jul 16 12:00:28.662873 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:28.662817    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk" podStartSLOduration=1.6627995009999998 podStartE2EDuration="1.662799501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:00:28.660714606 +0000 UTC m=+569.961491705" watchObservedRunningTime="2026-07-16 12:00:28.662799501 +0000 UTC m=+569.963576769"
Jul 16 12:00:28.827830 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:28.827788    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:29.832396 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:29.832356    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:30.611104 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:30.611068    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:30.611939 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:30.611918    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:00:30.914882 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:30.914809    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-lswfr"]
Jul 16 12:00:30.948266 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:30.948239    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-lswfr"]
Jul 16 12:00:30.948414 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:30.948322    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-lswfr"
Jul 16 12:00:30.951297 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:30.951265    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vjwgt\""
Jul 16 12:00:31.118895 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:31.118867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45zj8\" (UniqueName: \"kubernetes.io/projected/e576b36e-65eb-47d1-bec1-64d485ca6f27-kube-api-access-45zj8\") pod \"authorino-58cd955675-lswfr\" (UID: \"e576b36e-65eb-47d1-bec1-64d485ca6f27\") " pod="kuadrant-system/authorino-58cd955675-lswfr"
Jul 16 12:00:31.219896 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:31.219832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45zj8\" (UniqueName: \"kubernetes.io/projected/e576b36e-65eb-47d1-bec1-64d485ca6f27-kube-api-access-45zj8\") pod \"authorino-58cd955675-lswfr\" (UID: \"e576b36e-65eb-47d1-bec1-64d485ca6f27\") " pod="kuadrant-system/authorino-58cd955675-lswfr"
Jul 16 12:00:31.234305 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:31.234279    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45zj8\" (UniqueName: \"kubernetes.io/projected/e576b36e-65eb-47d1-bec1-64d485ca6f27-kube-api-access-45zj8\") pod \"authorino-58cd955675-lswfr\" (UID: \"e576b36e-65eb-47d1-bec1-64d485ca6f27\") " pod="kuadrant-system/authorino-58cd955675-lswfr"
Jul 16 12:00:31.258183 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:31.258159    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-lswfr"
Jul 16 12:00:31.418212 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:31.418187    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-lswfr"]
Jul 16 12:00:31.420212 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:00:31.420184    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode576b36e_65eb_47d1_bec1_64d485ca6f27.slice/crio-8d10e8a7defd6af61edda03cee33b6bc9a789b248cd93f6df648c1ccc910a31b WatchSource:0}: Error finding container 8d10e8a7defd6af61edda03cee33b6bc9a789b248cd93f6df648c1ccc910a31b: Status 404 returned error can't find the container with id 8d10e8a7defd6af61edda03cee33b6bc9a789b248cd93f6df648c1ccc910a31b
Jul 16 12:00:31.616789 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:31.616757    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-lswfr" event={"ID":"e576b36e-65eb-47d1-bec1-64d485ca6f27","Type":"ContainerStarted","Data":"8d10e8a7defd6af61edda03cee33b6bc9a789b248cd93f6df648c1ccc910a31b"}
Jul 16 12:00:35.641141 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:35.641098    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-lswfr" event={"ID":"e576b36e-65eb-47d1-bec1-64d485ca6f27","Type":"ContainerStarted","Data":"322cabed1fa7b11ee20f543f14523a2d358b3219cf7ee740ced859f609c9c664"}
Jul 16 12:00:35.657748 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:35.657695    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-lswfr" podStartSLOduration=2.411627158 podStartE2EDuration="5.657676176s" totalImagesPullingTime="3.246049018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:00:35.656720402 +0000 UTC m=+576.957497501" watchObservedRunningTime="2026-07-16 12:00:35.657676176 +0000 UTC m=+576.958453274"
Jul 16 12:00:37.666592 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:37.666554    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-lswfr"]
Jul 16 12:00:37.667004 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:37.666731    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-lswfr" podUID="e576b36e-65eb-47d1-bec1-64d485ca6f27" containerName="authorino" containerID="cri-o://322cabed1fa7b11ee20f543f14523a2d358b3219cf7ee740ced859f609c9c664" gracePeriod=30
Jul 16 12:00:37.924600 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:37.924540    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-lswfr"
Jul 16 12:00:38.080422 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:38.080385    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45zj8\" (UniqueName: \"kubernetes.io/projected/e576b36e-65eb-47d1-bec1-64d485ca6f27-kube-api-access-45zj8\") pod \"e576b36e-65eb-47d1-bec1-64d485ca6f27\" (UID: \"e576b36e-65eb-47d1-bec1-64d485ca6f27\") "
Jul 16 12:00:38.082482 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:38.082456    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e576b36e-65eb-47d1-bec1-64d485ca6f27-kube-api-access-45zj8" (OuterVolumeSpecName: "kube-api-access-45zj8") pod "e576b36e-65eb-47d1-bec1-64d485ca6f27" (UID: "e576b36e-65eb-47d1-bec1-64d485ca6f27"). InnerVolumeSpecName "kube-api-access-45zj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:00:38.181898 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:38.181825    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45zj8\" (UniqueName: \"kubernetes.io/projected/e576b36e-65eb-47d1-bec1-64d485ca6f27-kube-api-access-45zj8\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:00:38.656823 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:38.656793    2578 generic.go:356] "Generic (PLEG): container finished" podID="e576b36e-65eb-47d1-bec1-64d485ca6f27" containerID="322cabed1fa7b11ee20f543f14523a2d358b3219cf7ee740ced859f609c9c664" exitCode=0
Jul 16 12:00:38.657032 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:38.656849    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-lswfr"
Jul 16 12:00:38.657032 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:38.656852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-lswfr" event={"ID":"e576b36e-65eb-47d1-bec1-64d485ca6f27","Type":"ContainerDied","Data":"322cabed1fa7b11ee20f543f14523a2d358b3219cf7ee740ced859f609c9c664"}
Jul 16 12:00:38.657032 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:38.656969    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-lswfr" event={"ID":"e576b36e-65eb-47d1-bec1-64d485ca6f27","Type":"ContainerDied","Data":"8d10e8a7defd6af61edda03cee33b6bc9a789b248cd93f6df648c1ccc910a31b"}
Jul 16 12:00:38.657032 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:38.656988    2578 scope.go:117] "RemoveContainer" containerID="322cabed1fa7b11ee20f543f14523a2d358b3219cf7ee740ced859f609c9c664"
Jul 16 12:00:38.665889 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:38.665872    2578 scope.go:117] "RemoveContainer" containerID="322cabed1fa7b11ee20f543f14523a2d358b3219cf7ee740ced859f609c9c664"
Jul 16 12:00:38.666134 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:00:38.666116    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"322cabed1fa7b11ee20f543f14523a2d358b3219cf7ee740ced859f609c9c664\": container with ID starting with 322cabed1fa7b11ee20f543f14523a2d358b3219cf7ee740ced859f609c9c664 not found: ID does not exist" containerID="322cabed1fa7b11ee20f543f14523a2d358b3219cf7ee740ced859f609c9c664"
Jul 16 12:00:38.666188 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:38.666143    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322cabed1fa7b11ee20f543f14523a2d358b3219cf7ee740ced859f609c9c664"} err="failed to get container status \"322cabed1fa7b11ee20f543f14523a2d358b3219cf7ee740ced859f609c9c664\": rpc error: code = NotFound desc = could not find container \"322cabed1fa7b11ee20f543f14523a2d358b3219cf7ee740ced859f609c9c664\": container with ID starting with 322cabed1fa7b11ee20f543f14523a2d358b3219cf7ee740ced859f609c9c664 not found: ID does not exist"
Jul 16 12:00:38.682449 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:38.682427    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-lswfr"]
Jul 16 12:00:38.689235 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:38.689215    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-lswfr"]
Jul 16 12:00:39.299350 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:00:39.299324    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e576b36e-65eb-47d1-bec1-64d485ca6f27" path="/var/lib/kubelet/pods/e576b36e-65eb-47d1-bec1-64d485ca6f27/volumes"
Jul 16 12:01:04.718238 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:04.718208    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4hd8s"]
Jul 16 12:01:04.718691 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:04.718533    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e576b36e-65eb-47d1-bec1-64d485ca6f27" containerName="authorino"
Jul 16 12:01:04.718691 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:04.718545    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e576b36e-65eb-47d1-bec1-64d485ca6f27" containerName="authorino"
Jul 16 12:01:04.718691 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:04.718604    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e576b36e-65eb-47d1-bec1-64d485ca6f27" containerName="authorino"
Jul 16 12:01:04.721424 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:04.721405    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4hd8s"
Jul 16 12:01:04.723748 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:04.723720    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vjwgt\""
Jul 16 12:01:04.729294 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:04.729268    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4hd8s"]
Jul 16 12:01:04.790101 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:04.790071    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2svjz\" (UniqueName: \"kubernetes.io/projected/5aff4955-3e61-4e1a-9f96-4cf97bf3c63f-kube-api-access-2svjz\") pod \"authorino-5d4b67c446-4hd8s\" (UID: \"5aff4955-3e61-4e1a-9f96-4cf97bf3c63f\") " pod="kuadrant-system/authorino-5d4b67c446-4hd8s"
Jul 16 12:01:04.891205 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:04.891166    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2svjz\" (UniqueName: \"kubernetes.io/projected/5aff4955-3e61-4e1a-9f96-4cf97bf3c63f-kube-api-access-2svjz\") pod \"authorino-5d4b67c446-4hd8s\" (UID: \"5aff4955-3e61-4e1a-9f96-4cf97bf3c63f\") " pod="kuadrant-system/authorino-5d4b67c446-4hd8s"
Jul 16 12:01:04.901265 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:04.901243    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2svjz\" (UniqueName: \"kubernetes.io/projected/5aff4955-3e61-4e1a-9f96-4cf97bf3c63f-kube-api-access-2svjz\") pod \"authorino-5d4b67c446-4hd8s\" (UID: \"5aff4955-3e61-4e1a-9f96-4cf97bf3c63f\") " pod="kuadrant-system/authorino-5d4b67c446-4hd8s"
Jul 16 12:01:04.934836 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:04.934804    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4hd8s"]
Jul 16 12:01:04.935077 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:04.935065    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4hd8s"
Jul 16 12:01:04.966237 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:04.966210    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-665678d975-hj5tm"]
Jul 16 12:01:04.969209 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:04.969164    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-665678d975-hj5tm"
Jul 16 12:01:04.973366 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:04.973337    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 16 12:01:04.982335 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:04.982309    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-665678d975-hj5tm"]
Jul 16 12:01:04.992311 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:04.992278    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv6w8\" (UniqueName: \"kubernetes.io/projected/9feb1743-d4d1-46a1-b0ff-2330ffdbc624-kube-api-access-pv6w8\") pod \"authorino-665678d975-hj5tm\" (UID: \"9feb1743-d4d1-46a1-b0ff-2330ffdbc624\") " pod="kuadrant-system/authorino-665678d975-hj5tm"
Jul 16 12:01:04.992447 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:04.992424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9feb1743-d4d1-46a1-b0ff-2330ffdbc624-tls-cert\") pod \"authorino-665678d975-hj5tm\" (UID: \"9feb1743-d4d1-46a1-b0ff-2330ffdbc624\") " pod="kuadrant-system/authorino-665678d975-hj5tm"
Jul 16 12:01:05.067397 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:05.067361    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4hd8s"]
Jul 16 12:01:05.070393 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:01:05.070368    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5aff4955_3e61_4e1a_9f96_4cf97bf3c63f.slice/crio-1b7775e3d29f611b96ce878555253775735e907480d1d9f56c84cd5898efcb65 WatchSource:0}: Error finding container 1b7775e3d29f611b96ce878555253775735e907480d1d9f56c84cd5898efcb65: Status 404 returned error can't find the container with id 1b7775e3d29f611b96ce878555253775735e907480d1d9f56c84cd5898efcb65
Jul 16 12:01:05.093055 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:05.093021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9feb1743-d4d1-46a1-b0ff-2330ffdbc624-tls-cert\") pod \"authorino-665678d975-hj5tm\" (UID: \"9feb1743-d4d1-46a1-b0ff-2330ffdbc624\") " pod="kuadrant-system/authorino-665678d975-hj5tm"
Jul 16 12:01:05.093152 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:05.093109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv6w8\" (UniqueName: \"kubernetes.io/projected/9feb1743-d4d1-46a1-b0ff-2330ffdbc624-kube-api-access-pv6w8\") pod \"authorino-665678d975-hj5tm\" (UID: \"9feb1743-d4d1-46a1-b0ff-2330ffdbc624\") " pod="kuadrant-system/authorino-665678d975-hj5tm"
Jul 16 12:01:05.095279 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:05.095253    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9feb1743-d4d1-46a1-b0ff-2330ffdbc624-tls-cert\") pod \"authorino-665678d975-hj5tm\" (UID: \"9feb1743-d4d1-46a1-b0ff-2330ffdbc624\") " pod="kuadrant-system/authorino-665678d975-hj5tm"
Jul 16 12:01:05.104270 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:05.104253    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv6w8\" (UniqueName: \"kubernetes.io/projected/9feb1743-d4d1-46a1-b0ff-2330ffdbc624-kube-api-access-pv6w8\") pod \"authorino-665678d975-hj5tm\" (UID: \"9feb1743-d4d1-46a1-b0ff-2330ffdbc624\") " pod="kuadrant-system/authorino-665678d975-hj5tm"
Jul 16 12:01:05.280226 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:05.280145    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-665678d975-hj5tm"
Jul 16 12:01:05.413062 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:05.413020    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-665678d975-hj5tm"]
Jul 16 12:01:05.415389 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:01:05.415363    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9feb1743_d4d1_46a1_b0ff_2330ffdbc624.slice/crio-0b5906c07c6c527028e7ae4e510c8ab3337445a71cdc8c3aa5762b5fb3d427fb WatchSource:0}: Error finding container 0b5906c07c6c527028e7ae4e510c8ab3337445a71cdc8c3aa5762b5fb3d427fb: Status 404 returned error can't find the container with id 0b5906c07c6c527028e7ae4e510c8ab3337445a71cdc8c3aa5762b5fb3d427fb
Jul 16 12:01:05.787555 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:05.787528    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-665678d975-hj5tm" event={"ID":"9feb1743-d4d1-46a1-b0ff-2330ffdbc624","Type":"ContainerStarted","Data":"0b5906c07c6c527028e7ae4e510c8ab3337445a71cdc8c3aa5762b5fb3d427fb"}
Jul 16 12:01:05.788937 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:05.788911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-4hd8s" event={"ID":"5aff4955-3e61-4e1a-9f96-4cf97bf3c63f","Type":"ContainerStarted","Data":"e066e54a202d1b22a75155f85ee2279e7600fcb80f6a19f0ea6b28b567e454a2"}
Jul 16 12:01:05.789052 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:05.788948    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-4hd8s" event={"ID":"5aff4955-3e61-4e1a-9f96-4cf97bf3c63f","Type":"ContainerStarted","Data":"1b7775e3d29f611b96ce878555253775735e907480d1d9f56c84cd5898efcb65"}
Jul 16 12:01:05.789052 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:05.788955    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-4hd8s" podUID="5aff4955-3e61-4e1a-9f96-4cf97bf3c63f" containerName="authorino" containerID="cri-o://e066e54a202d1b22a75155f85ee2279e7600fcb80f6a19f0ea6b28b567e454a2" gracePeriod=30
Jul 16 12:01:05.807379 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:05.807329    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-4hd8s" podStartSLOduration=1.367025586 podStartE2EDuration="1.807307498s" totalImagesPullingTime="440.281912ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:01:05.805268165 +0000 UTC m=+607.106045264" watchObservedRunningTime="2026-07-16 12:01:05.807307498 +0000 UTC m=+607.108084602"
Jul 16 12:01:06.029556 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:06.029532    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4hd8s"
Jul 16 12:01:06.101536 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:06.101479    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2svjz\" (UniqueName: \"kubernetes.io/projected/5aff4955-3e61-4e1a-9f96-4cf97bf3c63f-kube-api-access-2svjz\") pod \"5aff4955-3e61-4e1a-9f96-4cf97bf3c63f\" (UID: \"5aff4955-3e61-4e1a-9f96-4cf97bf3c63f\") "
Jul 16 12:01:06.103468 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:06.103446    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aff4955-3e61-4e1a-9f96-4cf97bf3c63f-kube-api-access-2svjz" (OuterVolumeSpecName: "kube-api-access-2svjz") pod "5aff4955-3e61-4e1a-9f96-4cf97bf3c63f" (UID: "5aff4955-3e61-4e1a-9f96-4cf97bf3c63f"). InnerVolumeSpecName "kube-api-access-2svjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:01:06.202198 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:06.202170    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2svjz\" (UniqueName: \"kubernetes.io/projected/5aff4955-3e61-4e1a-9f96-4cf97bf3c63f-kube-api-access-2svjz\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:01:06.795098 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:06.795066    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-665678d975-hj5tm" event={"ID":"9feb1743-d4d1-46a1-b0ff-2330ffdbc624","Type":"ContainerStarted","Data":"99e24fd5b2bc87829aeb5137072f792191b10f170f665be8be1f3beba388bc10"}
Jul 16 12:01:06.796302 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:06.796281    2578 generic.go:356] "Generic (PLEG): container finished" podID="5aff4955-3e61-4e1a-9f96-4cf97bf3c63f" containerID="e066e54a202d1b22a75155f85ee2279e7600fcb80f6a19f0ea6b28b567e454a2" exitCode=0
Jul 16 12:01:06.796423 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:06.796318    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4hd8s"
Jul 16 12:01:06.796423 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:06.796345    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-4hd8s" event={"ID":"5aff4955-3e61-4e1a-9f96-4cf97bf3c63f","Type":"ContainerDied","Data":"e066e54a202d1b22a75155f85ee2279e7600fcb80f6a19f0ea6b28b567e454a2"}
Jul 16 12:01:06.796423 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:06.796367    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-4hd8s" event={"ID":"5aff4955-3e61-4e1a-9f96-4cf97bf3c63f","Type":"ContainerDied","Data":"1b7775e3d29f611b96ce878555253775735e907480d1d9f56c84cd5898efcb65"}
Jul 16 12:01:06.796423 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:06.796383    2578 scope.go:117] "RemoveContainer" containerID="e066e54a202d1b22a75155f85ee2279e7600fcb80f6a19f0ea6b28b567e454a2"
Jul 16 12:01:06.804928 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:06.804913    2578 scope.go:117] "RemoveContainer" containerID="e066e54a202d1b22a75155f85ee2279e7600fcb80f6a19f0ea6b28b567e454a2"
Jul 16 12:01:06.805181 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:01:06.805165    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e066e54a202d1b22a75155f85ee2279e7600fcb80f6a19f0ea6b28b567e454a2\": container with ID starting with e066e54a202d1b22a75155f85ee2279e7600fcb80f6a19f0ea6b28b567e454a2 not found: ID does not exist" containerID="e066e54a202d1b22a75155f85ee2279e7600fcb80f6a19f0ea6b28b567e454a2"
Jul 16 12:01:06.805231 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:06.805188    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e066e54a202d1b22a75155f85ee2279e7600fcb80f6a19f0ea6b28b567e454a2"} err="failed to get container status \"e066e54a202d1b22a75155f85ee2279e7600fcb80f6a19f0ea6b28b567e454a2\": rpc error: code = NotFound desc = could not find container \"e066e54a202d1b22a75155f85ee2279e7600fcb80f6a19f0ea6b28b567e454a2\": container with ID starting with e066e54a202d1b22a75155f85ee2279e7600fcb80f6a19f0ea6b28b567e454a2 not found: ID does not exist"
Jul 16 12:01:06.821930 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:06.821892    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-665678d975-hj5tm" podStartSLOduration=2.516056073 podStartE2EDuration="2.821879696s" totalImagesPullingTime="305.823623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:01:06.821208321 +0000 UTC m=+608.121985421" watchObservedRunningTime="2026-07-16 12:01:06.821879696 +0000 UTC m=+608.122656793"
Jul 16 12:01:06.845912 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:06.845891    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4hd8s"]
Jul 16 12:01:06.857110 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:06.857089    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4hd8s"]
Jul 16 12:01:07.299144 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:01:07.299115    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5aff4955-3e61-4e1a-9f96-4cf97bf3c63f" path="/var/lib/kubelet/pods/5aff4955-3e61-4e1a-9f96-4cf97bf3c63f/volumes"
Jul 16 12:02:00.874996 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:00.874420    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-bzksk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": read tcp 10.133.0.2:37042->10.133.0.35:15021: read: connection reset by peer" start-of-body=
Jul 16 12:02:00.874996 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:00.874487    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk" podUID="2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": read tcp 10.133.0.2:37042->10.133.0.35:15021: read: connection reset by peer"
Jul 16 12:02:01.052430 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:01.052402    2578 generic.go:356] "Generic (PLEG): container finished" podID="2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421" containerID="b4aa654b234f0db424113ab4f111ac9f674f452d607a494a0ef15ddbc7600fbb" exitCode=0
Jul 16 12:02:01.052544 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:01.052465    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk" event={"ID":"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421","Type":"ContainerDied","Data":"b4aa654b234f0db424113ab4f111ac9f674f452d607a494a0ef15ddbc7600fbb"}
Jul 16 12:02:01.052790 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:01.052778    2578 scope.go:117] "RemoveContainer" containerID="b4aa654b234f0db424113ab4f111ac9f674f452d607a494a0ef15ddbc7600fbb"
Jul 16 12:02:01.053712 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:01.053684    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:02:01.053769 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:01.053755    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:02:01.053805 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:01.053792    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:02:02.060173 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:02.060138    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk" event={"ID":"2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421","Type":"ContainerStarted","Data":"23180e8366f1470af5c2acef277906aecacf75d1adf4400398bbe2bcc6d50599"}
Jul 16 12:02:02.828017 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:02.827976    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:02:02.828300 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:02.828270    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-bzksk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 16 12:02:02.828384 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:02.828350    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk" podUID="2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 16 12:02:03.827809 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:03.827373    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-bzksk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 16 12:02:03.827809 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:03.827431    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk" podUID="2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 16 12:02:04.828385 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:04.828344    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-bzksk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 16 12:02:04.828870 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:04.828417    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk" podUID="2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 16 12:02:05.827608 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:05.827572    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-bzksk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 16 12:02:05.827908 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:05.827881    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk" podUID="2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 16 12:02:06.831639 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:06.831609    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:02:06.832085 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:06.831988    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:02:06.832691 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:02:06.832668    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bzksk"
Jul 16 12:03:00.524503 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.524474    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"]
Jul 16 12:03:00.525119 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.524886    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aff4955-3e61-4e1a-9f96-4cf97bf3c63f" containerName="authorino"
Jul 16 12:03:00.525119 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.524902    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aff4955-3e61-4e1a-9f96-4cf97bf3c63f" containerName="authorino"
Jul 16 12:03:00.525119 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.524975    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5aff4955-3e61-4e1a-9f96-4cf97bf3c63f" containerName="authorino"
Jul 16 12:03:00.527019 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.527000    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:00.535325 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.535301    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 16 12:03:00.535430 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.535336    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 16 12:03:00.535430 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.535383    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6577m\""
Jul 16 12:03:00.535430 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.535391    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 16 12:03:00.546483 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.546463    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"]
Jul 16 12:03:00.622868 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.622844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3906585-214d-4a81-914c-3f2807b41cfa-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pc6bk\" (UID: \"e3906585-214d-4a81-914c-3f2807b41cfa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:00.622987 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.622877    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3906585-214d-4a81-914c-3f2807b41cfa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pc6bk\" (UID: \"e3906585-214d-4a81-914c-3f2807b41cfa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:00.622987 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.622895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3906585-214d-4a81-914c-3f2807b41cfa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pc6bk\" (UID: \"e3906585-214d-4a81-914c-3f2807b41cfa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:00.622987 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.622968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrxlt\" (UniqueName: \"kubernetes.io/projected/e3906585-214d-4a81-914c-3f2807b41cfa-kube-api-access-lrxlt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pc6bk\" (UID: \"e3906585-214d-4a81-914c-3f2807b41cfa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:00.623195 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.623064    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3906585-214d-4a81-914c-3f2807b41cfa-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pc6bk\" (UID: \"e3906585-214d-4a81-914c-3f2807b41cfa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:00.623195 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.623105    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3906585-214d-4a81-914c-3f2807b41cfa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pc6bk\" (UID: \"e3906585-214d-4a81-914c-3f2807b41cfa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:00.724338 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.724301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3906585-214d-4a81-914c-3f2807b41cfa-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pc6bk\" (UID: \"e3906585-214d-4a81-914c-3f2807b41cfa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:00.724474 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.724354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3906585-214d-4a81-914c-3f2807b41cfa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pc6bk\" (UID: \"e3906585-214d-4a81-914c-3f2807b41cfa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:00.724474 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.724408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3906585-214d-4a81-914c-3f2807b41cfa-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pc6bk\" (UID: \"e3906585-214d-4a81-914c-3f2807b41cfa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:00.724474 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.724444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3906585-214d-4a81-914c-3f2807b41cfa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pc6bk\" (UID: \"e3906585-214d-4a81-914c-3f2807b41cfa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:00.724474 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.724466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3906585-214d-4a81-914c-3f2807b41cfa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pc6bk\" (UID: \"e3906585-214d-4a81-914c-3f2807b41cfa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:00.724676 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.724516    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrxlt\" (UniqueName: \"kubernetes.io/projected/e3906585-214d-4a81-914c-3f2807b41cfa-kube-api-access-lrxlt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pc6bk\" (UID: \"e3906585-214d-4a81-914c-3f2807b41cfa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:00.724773 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.724750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3906585-214d-4a81-914c-3f2807b41cfa-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pc6bk\" (UID: \"e3906585-214d-4a81-914c-3f2807b41cfa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:00.724874 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.724855    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3906585-214d-4a81-914c-3f2807b41cfa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pc6bk\" (UID: \"e3906585-214d-4a81-914c-3f2807b41cfa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:00.725093 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.725027    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3906585-214d-4a81-914c-3f2807b41cfa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pc6bk\" (UID: \"e3906585-214d-4a81-914c-3f2807b41cfa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:00.726708 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.726687    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3906585-214d-4a81-914c-3f2807b41cfa-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pc6bk\" (UID: \"e3906585-214d-4a81-914c-3f2807b41cfa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:00.727261 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.727243    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3906585-214d-4a81-914c-3f2807b41cfa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pc6bk\" (UID: \"e3906585-214d-4a81-914c-3f2807b41cfa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:00.735164 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.735138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrxlt\" (UniqueName: \"kubernetes.io/projected/e3906585-214d-4a81-914c-3f2807b41cfa-kube-api-access-lrxlt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pc6bk\" (UID: \"e3906585-214d-4a81-914c-3f2807b41cfa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:00.837178 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.837148    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:00.970222 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.970199    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"]
Jul 16 12:03:00.971878 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:03:00.971848    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3906585_214d_4a81_914c_3f2807b41cfa.slice/crio-f805b40ab5952e3ebee9e3a6974f15299497dbbc816bb5bafb3b6f2dfa7e533c WatchSource:0}: Error finding container f805b40ab5952e3ebee9e3a6974f15299497dbbc816bb5bafb3b6f2dfa7e533c: Status 404 returned error can't find the container with id f805b40ab5952e3ebee9e3a6974f15299497dbbc816bb5bafb3b6f2dfa7e533c
Jul 16 12:03:00.973129 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:00.973112    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 12:03:01.338873 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:01.338804    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk" event={"ID":"e3906585-214d-4a81-914c-3f2807b41cfa","Type":"ContainerStarted","Data":"f805b40ab5952e3ebee9e3a6974f15299497dbbc816bb5bafb3b6f2dfa7e533c"}
Jul 16 12:03:02.346470 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:02.346434    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk" event={"ID":"e3906585-214d-4a81-914c-3f2807b41cfa","Type":"ContainerStarted","Data":"0d86d556a22581096802033e9d1bcc0c1395d7f62720ad785dd58c1ae1324a19"}
Jul 16 12:03:02.367368 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:02.367319    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk" podStartSLOduration=1.319599604 podStartE2EDuration="2.367302791s" totalImagesPullingTime="1.047703187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:03:02.365861259 +0000 UTC m=+723.666638370" watchObservedRunningTime="2026-07-16 12:03:02.367302791 +0000 UTC m=+723.668079888"
Jul 16 12:03:10.837766 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:10.837728    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:10.838194 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:10.837837    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:10.849868 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:10.849847    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:11.397200 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:11.397166    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pc6bk"
Jul 16 12:03:15.903510 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:15.903479    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"]
Jul 16 12:03:15.905816 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:15.905800    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:15.908398 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:15.908375    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 16 12:03:15.917754 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:15.917729    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"]
Jul 16 12:03:15.946890 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:15.946861    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t\" (UID: \"da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:15.947004 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:15.946894    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t\" (UID: \"da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:15.947004 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:15.946923    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mts4n\" (UniqueName: \"kubernetes.io/projected/da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820-kube-api-access-mts4n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t\" (UID: \"da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:15.947136 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:15.947050    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t\" (UID: \"da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:15.947136 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:15.947115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t\" (UID: \"da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:15.947205 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:15.947174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t\" (UID: \"da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:16.047954 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:16.047923    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mts4n\" (UniqueName: \"kubernetes.io/projected/da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820-kube-api-access-mts4n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t\" (UID: \"da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:16.048110 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:16.047958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t\" (UID: \"da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:16.048110 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:16.048089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t\" (UID: \"da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:16.048197 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:16.048139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t\" (UID: \"da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:16.048197 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:16.048185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t\" (UID: \"da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:16.048272 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:16.048204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t\" (UID: \"da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:16.048441 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:16.048422    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t\" (UID: \"da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:16.048504 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:16.048450    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t\" (UID: \"da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:16.048565 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:16.048540    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t\" (UID: \"da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:16.050242 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:16.050220    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t\" (UID: \"da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:16.050557 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:16.050535    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t\" (UID: \"da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:16.063596 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:16.063575    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mts4n\" (UniqueName: \"kubernetes.io/projected/da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820-kube-api-access-mts4n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t\" (UID: \"da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:16.216355 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:16.216282    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:16.355175 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:16.355135    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"]
Jul 16 12:03:16.357446 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:03:16.357418    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4d20e4_6edf_43e3_ae6c_ccc3aaa5b820.slice/crio-9a5a081e660f5fb9d3452a74c80331a4be54cf278d7914ca4c9c62663780dbf0 WatchSource:0}: Error finding container 9a5a081e660f5fb9d3452a74c80331a4be54cf278d7914ca4c9c62663780dbf0: Status 404 returned error can't find the container with id 9a5a081e660f5fb9d3452a74c80331a4be54cf278d7914ca4c9c62663780dbf0
Jul 16 12:03:16.408706 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:16.408679    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t" event={"ID":"da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820","Type":"ContainerStarted","Data":"9a5a081e660f5fb9d3452a74c80331a4be54cf278d7914ca4c9c62663780dbf0"}
Jul 16 12:03:17.414191 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:17.414156    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t" event={"ID":"da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820","Type":"ContainerStarted","Data":"c8f3914d12e6b3f86ca67c5f77271c914c2288d405e164d3e69d44a58e1f4f63"}
Jul 16 12:03:17.450008 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:17.449958    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t" podStartSLOduration=2.256343034 podStartE2EDuration="2.449944088s" totalImagesPullingTime="193.601054ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:03:17.448754402 +0000 UTC m=+738.749531500" watchObservedRunningTime="2026-07-16 12:03:17.449944088 +0000 UTC m=+738.750721186"
Jul 16 12:03:24.806163 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:24.806132    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"]
Jul 16 12:03:24.808513 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:24.808488    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:24.811555 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:24.811534    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 16 12:03:24.826957 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:24.826929    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"]
Jul 16 12:03:24.924469 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:24.924429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0866ac5e-6a94-4de0-85ec-e120a84159cc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n27gv\" (UID: \"0866ac5e-6a94-4de0-85ec-e120a84159cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:24.924613 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:24.924510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0866ac5e-6a94-4de0-85ec-e120a84159cc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n27gv\" (UID: \"0866ac5e-6a94-4de0-85ec-e120a84159cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:24.924613 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:24.924544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0866ac5e-6a94-4de0-85ec-e120a84159cc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n27gv\" (UID: \"0866ac5e-6a94-4de0-85ec-e120a84159cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:24.924613 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:24.924587    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0866ac5e-6a94-4de0-85ec-e120a84159cc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n27gv\" (UID: \"0866ac5e-6a94-4de0-85ec-e120a84159cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:24.924811 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:24.924621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0866ac5e-6a94-4de0-85ec-e120a84159cc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n27gv\" (UID: \"0866ac5e-6a94-4de0-85ec-e120a84159cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:24.924811 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:24.924652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qn6d\" (UniqueName: \"kubernetes.io/projected/0866ac5e-6a94-4de0-85ec-e120a84159cc-kube-api-access-2qn6d\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n27gv\" (UID: \"0866ac5e-6a94-4de0-85ec-e120a84159cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:25.025338 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:25.025300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0866ac5e-6a94-4de0-85ec-e120a84159cc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n27gv\" (UID: \"0866ac5e-6a94-4de0-85ec-e120a84159cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:25.025500 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:25.025352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0866ac5e-6a94-4de0-85ec-e120a84159cc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n27gv\" (UID: \"0866ac5e-6a94-4de0-85ec-e120a84159cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:25.025500 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:25.025404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0866ac5e-6a94-4de0-85ec-e120a84159cc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n27gv\" (UID: \"0866ac5e-6a94-4de0-85ec-e120a84159cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:25.025500 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:25.025441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0866ac5e-6a94-4de0-85ec-e120a84159cc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n27gv\" (UID: \"0866ac5e-6a94-4de0-85ec-e120a84159cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:25.025500 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:25.025475    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qn6d\" (UniqueName: \"kubernetes.io/projected/0866ac5e-6a94-4de0-85ec-e120a84159cc-kube-api-access-2qn6d\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n27gv\" (UID: \"0866ac5e-6a94-4de0-85ec-e120a84159cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:25.025701 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:25.025506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0866ac5e-6a94-4de0-85ec-e120a84159cc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n27gv\" (UID: \"0866ac5e-6a94-4de0-85ec-e120a84159cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:25.025758 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:25.025730    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0866ac5e-6a94-4de0-85ec-e120a84159cc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n27gv\" (UID: \"0866ac5e-6a94-4de0-85ec-e120a84159cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:25.026107 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:25.026083    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0866ac5e-6a94-4de0-85ec-e120a84159cc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n27gv\" (UID: \"0866ac5e-6a94-4de0-85ec-e120a84159cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:25.026315 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:25.026288    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0866ac5e-6a94-4de0-85ec-e120a84159cc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n27gv\" (UID: \"0866ac5e-6a94-4de0-85ec-e120a84159cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:25.028488 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:25.028461    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0866ac5e-6a94-4de0-85ec-e120a84159cc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n27gv\" (UID: \"0866ac5e-6a94-4de0-85ec-e120a84159cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:25.028620 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:25.028598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0866ac5e-6a94-4de0-85ec-e120a84159cc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n27gv\" (UID: \"0866ac5e-6a94-4de0-85ec-e120a84159cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:25.040657 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:25.040635    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qn6d\" (UniqueName: \"kubernetes.io/projected/0866ac5e-6a94-4de0-85ec-e120a84159cc-kube-api-access-2qn6d\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n27gv\" (UID: \"0866ac5e-6a94-4de0-85ec-e120a84159cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:25.119921 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:25.119842    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:25.257077 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:25.257020    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"]
Jul 16 12:03:25.452458 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:25.452377    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv" event={"ID":"0866ac5e-6a94-4de0-85ec-e120a84159cc","Type":"ContainerStarted","Data":"9fa70429dca0897c4dea78e6fbb393eaad36e47fd4c7917fe05565391c7e6a14"}
Jul 16 12:03:26.216523 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:26.216488    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:26.216913 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:26.216592    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:26.230967 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:26.230937    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:26.459463 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:26.459416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv" event={"ID":"0866ac5e-6a94-4de0-85ec-e120a84159cc","Type":"ContainerStarted","Data":"bd75989251cc3df7e08777887fe341703f600629b66ae62963f75d50848a705a"}
Jul 16 12:03:26.470716 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:26.470646    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t"
Jul 16 12:03:26.479409 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:26.479339    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv" podStartSLOduration=2.189464397 podStartE2EDuration="2.47932451s" totalImagesPullingTime="289.860113ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:03:26.478449724 +0000 UTC m=+747.779226819" watchObservedRunningTime="2026-07-16 12:03:26.47932451 +0000 UTC m=+747.780101607"
Jul 16 12:03:35.121080 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:35.121025    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:35.121521 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:35.121108    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:35.133237 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:35.133214    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:03:35.512877 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:03:35.512799    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n27gv"
Jul 16 12:04:22.832516 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:04:22.832436    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-665678d975-hj5tm"]
Jul 16 12:04:22.833012 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:04:22.832633    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-665678d975-hj5tm" podUID="9feb1743-d4d1-46a1-b0ff-2330ffdbc624" containerName="authorino" containerID="cri-o://99e24fd5b2bc87829aeb5137072f792191b10f170f665be8be1f3beba388bc10" gracePeriod=30
Jul 16 12:04:23.091663 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:04:23.091607    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-665678d975-hj5tm"
Jul 16 12:04:23.208732 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:04:23.208701    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv6w8\" (UniqueName: \"kubernetes.io/projected/9feb1743-d4d1-46a1-b0ff-2330ffdbc624-kube-api-access-pv6w8\") pod \"9feb1743-d4d1-46a1-b0ff-2330ffdbc624\" (UID: \"9feb1743-d4d1-46a1-b0ff-2330ffdbc624\") "
Jul 16 12:04:23.208897 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:04:23.208787    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9feb1743-d4d1-46a1-b0ff-2330ffdbc624-tls-cert\") pod \"9feb1743-d4d1-46a1-b0ff-2330ffdbc624\" (UID: \"9feb1743-d4d1-46a1-b0ff-2330ffdbc624\") "
Jul 16 12:04:23.210823 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:04:23.210790    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9feb1743-d4d1-46a1-b0ff-2330ffdbc624-kube-api-access-pv6w8" (OuterVolumeSpecName: "kube-api-access-pv6w8") pod "9feb1743-d4d1-46a1-b0ff-2330ffdbc624" (UID: "9feb1743-d4d1-46a1-b0ff-2330ffdbc624"). InnerVolumeSpecName "kube-api-access-pv6w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:04:23.219133 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:04:23.219106    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9feb1743-d4d1-46a1-b0ff-2330ffdbc624-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9feb1743-d4d1-46a1-b0ff-2330ffdbc624" (UID: "9feb1743-d4d1-46a1-b0ff-2330ffdbc624"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 12:04:23.309361 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:04:23.309336    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9feb1743-d4d1-46a1-b0ff-2330ffdbc624-tls-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:04:23.309448 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:04:23.309364    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv6w8\" (UniqueName: \"kubernetes.io/projected/9feb1743-d4d1-46a1-b0ff-2330ffdbc624-kube-api-access-pv6w8\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:04:23.728057 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:04:23.728001    2578 generic.go:356] "Generic (PLEG): container finished" podID="9feb1743-d4d1-46a1-b0ff-2330ffdbc624" containerID="99e24fd5b2bc87829aeb5137072f792191b10f170f665be8be1f3beba388bc10" exitCode=0
Jul 16 12:04:23.728247 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:04:23.728068    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-665678d975-hj5tm"
Jul 16 12:04:23.728247 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:04:23.728095    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-665678d975-hj5tm" event={"ID":"9feb1743-d4d1-46a1-b0ff-2330ffdbc624","Type":"ContainerDied","Data":"99e24fd5b2bc87829aeb5137072f792191b10f170f665be8be1f3beba388bc10"}
Jul 16 12:04:23.728247 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:04:23.728133    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-665678d975-hj5tm" event={"ID":"9feb1743-d4d1-46a1-b0ff-2330ffdbc624","Type":"ContainerDied","Data":"0b5906c07c6c527028e7ae4e510c8ab3337445a71cdc8c3aa5762b5fb3d427fb"}
Jul 16 12:04:23.728247 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:04:23.728151    2578 scope.go:117] "RemoveContainer" containerID="99e24fd5b2bc87829aeb5137072f792191b10f170f665be8be1f3beba388bc10"
Jul 16 12:04:23.736486 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:04:23.736469    2578 scope.go:117] "RemoveContainer" containerID="99e24fd5b2bc87829aeb5137072f792191b10f170f665be8be1f3beba388bc10"
Jul 16 12:04:23.736726 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:04:23.736708    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99e24fd5b2bc87829aeb5137072f792191b10f170f665be8be1f3beba388bc10\": container with ID starting with 99e24fd5b2bc87829aeb5137072f792191b10f170f665be8be1f3beba388bc10 not found: ID does not exist" containerID="99e24fd5b2bc87829aeb5137072f792191b10f170f665be8be1f3beba388bc10"
Jul 16 12:04:23.736794 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:04:23.736733    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99e24fd5b2bc87829aeb5137072f792191b10f170f665be8be1f3beba388bc10"} err="failed to get container status \"99e24fd5b2bc87829aeb5137072f792191b10f170f665be8be1f3beba388bc10\": rpc error: code = NotFound desc = could not find container \"99e24fd5b2bc87829aeb5137072f792191b10f170f665be8be1f3beba388bc10\": container with ID starting with 99e24fd5b2bc87829aeb5137072f792191b10f170f665be8be1f3beba388bc10 not found: ID does not exist"
Jul 16 12:04:23.792966 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:04:23.792941    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-665678d975-hj5tm"]
Jul 16 12:04:23.805418 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:04:23.805391    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-665678d975-hj5tm"]
Jul 16 12:04:25.300800 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:04:25.300772    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9feb1743-d4d1-46a1-b0ff-2330ffdbc624" path="/var/lib/kubelet/pods/9feb1743-d4d1-46a1-b0ff-2330ffdbc624/volumes"
Jul 16 12:10:56.653237 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:10:56.653155    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p"]
Jul 16 12:10:56.653740 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:10:56.653386    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p" podUID="160effe3-5586-4035-bde2-a3ae65043966" containerName="manager" containerID="cri-o://fe9953a0befc2e9c20fc15f3ac4fbad6c27f52e6863574b40c428ec4c3785190" gracePeriod=10
Jul 16 12:11:00.511902 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:00.511878    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p"
Jul 16 12:11:00.578468 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:00.578377    2578 generic.go:356] "Generic (PLEG): container finished" podID="160effe3-5586-4035-bde2-a3ae65043966" containerID="fe9953a0befc2e9c20fc15f3ac4fbad6c27f52e6863574b40c428ec4c3785190" exitCode=0
Jul 16 12:11:00.578468 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:00.578424    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p"
Jul 16 12:11:00.578468 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:00.578441    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p" event={"ID":"160effe3-5586-4035-bde2-a3ae65043966","Type":"ContainerDied","Data":"fe9953a0befc2e9c20fc15f3ac4fbad6c27f52e6863574b40c428ec4c3785190"}
Jul 16 12:11:00.578661 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:00.578526    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p" event={"ID":"160effe3-5586-4035-bde2-a3ae65043966","Type":"ContainerDied","Data":"bf734c915ea5c3eedb0cd1f3825be32b82cf9522b34494cdfd0532a4a7c1b6d9"}
Jul 16 12:11:00.578661 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:00.578555    2578 scope.go:117] "RemoveContainer" containerID="fe9953a0befc2e9c20fc15f3ac4fbad6c27f52e6863574b40c428ec4c3785190"
Jul 16 12:11:00.588449 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:00.588423    2578 scope.go:117] "RemoveContainer" containerID="fe9953a0befc2e9c20fc15f3ac4fbad6c27f52e6863574b40c428ec4c3785190"
Jul 16 12:11:00.588712 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:11:00.588689    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe9953a0befc2e9c20fc15f3ac4fbad6c27f52e6863574b40c428ec4c3785190\": container with ID starting with fe9953a0befc2e9c20fc15f3ac4fbad6c27f52e6863574b40c428ec4c3785190 not found: ID does not exist" containerID="fe9953a0befc2e9c20fc15f3ac4fbad6c27f52e6863574b40c428ec4c3785190"
Jul 16 12:11:00.588776 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:00.588723    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9953a0befc2e9c20fc15f3ac4fbad6c27f52e6863574b40c428ec4c3785190"} err="failed to get container status \"fe9953a0befc2e9c20fc15f3ac4fbad6c27f52e6863574b40c428ec4c3785190\": rpc error: code = NotFound desc = could not find container \"fe9953a0befc2e9c20fc15f3ac4fbad6c27f52e6863574b40c428ec4c3785190\": container with ID starting with fe9953a0befc2e9c20fc15f3ac4fbad6c27f52e6863574b40c428ec4c3785190 not found: ID does not exist"
Jul 16 12:11:00.597987 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:00.597968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/160effe3-5586-4035-bde2-a3ae65043966-extensions-socket-volume\") pod \"160effe3-5586-4035-bde2-a3ae65043966\" (UID: \"160effe3-5586-4035-bde2-a3ae65043966\") "
Jul 16 12:11:00.598117 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:00.598106    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vz26j\" (UniqueName: \"kubernetes.io/projected/160effe3-5586-4035-bde2-a3ae65043966-kube-api-access-vz26j\") pod \"160effe3-5586-4035-bde2-a3ae65043966\" (UID: \"160effe3-5586-4035-bde2-a3ae65043966\") "
Jul 16 12:11:00.598370 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:00.598333    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160effe3-5586-4035-bde2-a3ae65043966-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "160effe3-5586-4035-bde2-a3ae65043966" (UID: "160effe3-5586-4035-bde2-a3ae65043966"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:11:00.599969 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:00.599945    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160effe3-5586-4035-bde2-a3ae65043966-kube-api-access-vz26j" (OuterVolumeSpecName: "kube-api-access-vz26j") pod "160effe3-5586-4035-bde2-a3ae65043966" (UID: "160effe3-5586-4035-bde2-a3ae65043966"). InnerVolumeSpecName "kube-api-access-vz26j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:11:00.699559 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:00.699529    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz26j\" (UniqueName: \"kubernetes.io/projected/160effe3-5586-4035-bde2-a3ae65043966-kube-api-access-vz26j\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:11:00.699559 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:00.699554    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/160effe3-5586-4035-bde2-a3ae65043966-extensions-socket-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:11:00.905979 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:00.905948    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p"]
Jul 16 12:11:00.919484 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:00.919459    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p"]
Jul 16 12:11:01.299613 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:01.299583    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="160effe3-5586-4035-bde2-a3ae65043966" path="/var/lib/kubelet/pods/160effe3-5586-4035-bde2-a3ae65043966/volumes"
Jul 16 12:11:01.469972 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:01.469935    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6nf2p" podUID="160effe3-5586-4035-bde2-a3ae65043966" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 16 12:11:57.951625 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:57.951549    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gtqx2"]
Jul 16 12:11:57.952097 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:57.951910    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9feb1743-d4d1-46a1-b0ff-2330ffdbc624" containerName="authorino"
Jul 16 12:11:57.952097 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:57.951921    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9feb1743-d4d1-46a1-b0ff-2330ffdbc624" containerName="authorino"
Jul 16 12:11:57.952097 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:57.951938    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="160effe3-5586-4035-bde2-a3ae65043966" containerName="manager"
Jul 16 12:11:57.952097 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:57.951943    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="160effe3-5586-4035-bde2-a3ae65043966" containerName="manager"
Jul 16 12:11:57.952097 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:57.952008    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9feb1743-d4d1-46a1-b0ff-2330ffdbc624" containerName="authorino"
Jul 16 12:11:57.952097 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:57.952019    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="160effe3-5586-4035-bde2-a3ae65043966" containerName="manager"
Jul 16 12:11:57.954834 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:57.954818    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gtqx2"
Jul 16 12:11:57.959081 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:57.959060    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fwlpm\""
Jul 16 12:11:57.987832 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:57.987807    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gtqx2"]
Jul 16 12:11:58.087170 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:58.087144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d30358f9-a62b-4fb1-8df4-374c0419d33a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gtqx2\" (UID: \"d30358f9-a62b-4fb1-8df4-374c0419d33a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gtqx2"
Jul 16 12:11:58.087275 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:58.087228    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b4pt\" (UniqueName: \"kubernetes.io/projected/d30358f9-a62b-4fb1-8df4-374c0419d33a-kube-api-access-2b4pt\") pod \"kuadrant-operator-controller-manager-86878c769f-gtqx2\" (UID: \"d30358f9-a62b-4fb1-8df4-374c0419d33a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gtqx2"
Jul 16 12:11:58.188023 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:58.187996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d30358f9-a62b-4fb1-8df4-374c0419d33a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gtqx2\" (UID: \"d30358f9-a62b-4fb1-8df4-374c0419d33a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gtqx2"
Jul 16 12:11:58.188168 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:58.188101    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b4pt\" (UniqueName: \"kubernetes.io/projected/d30358f9-a62b-4fb1-8df4-374c0419d33a-kube-api-access-2b4pt\") pod \"kuadrant-operator-controller-manager-86878c769f-gtqx2\" (UID: \"d30358f9-a62b-4fb1-8df4-374c0419d33a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gtqx2"
Jul 16 12:11:58.188348 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:58.188330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d30358f9-a62b-4fb1-8df4-374c0419d33a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gtqx2\" (UID: \"d30358f9-a62b-4fb1-8df4-374c0419d33a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gtqx2"
Jul 16 12:11:58.198689 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:58.198668    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b4pt\" (UniqueName: \"kubernetes.io/projected/d30358f9-a62b-4fb1-8df4-374c0419d33a-kube-api-access-2b4pt\") pod \"kuadrant-operator-controller-manager-86878c769f-gtqx2\" (UID: \"d30358f9-a62b-4fb1-8df4-374c0419d33a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gtqx2"
Jul 16 12:11:58.264666 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:58.264609    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gtqx2"
Jul 16 12:11:58.425555 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:58.425526    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gtqx2"]
Jul 16 12:11:58.429849 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:11:58.429814    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30358f9_a62b_4fb1_8df4_374c0419d33a.slice/crio-a93e7a99727f32112b78191ed4bf16a6acd2f0d3be54774feebc141e2d337f33 WatchSource:0}: Error finding container a93e7a99727f32112b78191ed4bf16a6acd2f0d3be54774feebc141e2d337f33: Status 404 returned error can't find the container with id a93e7a99727f32112b78191ed4bf16a6acd2f0d3be54774feebc141e2d337f33
Jul 16 12:11:58.852893 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:58.852861    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gtqx2" event={"ID":"d30358f9-a62b-4fb1-8df4-374c0419d33a","Type":"ContainerStarted","Data":"aeb1f34c011ba867746d30d6c39c48ca667d1a359e1fc3e3d4416974f1b569b1"}
Jul 16 12:11:58.853106 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:58.852911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gtqx2" event={"ID":"d30358f9-a62b-4fb1-8df4-374c0419d33a","Type":"ContainerStarted","Data":"a93e7a99727f32112b78191ed4bf16a6acd2f0d3be54774feebc141e2d337f33"}
Jul 16 12:11:58.853106 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:58.852973    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gtqx2"
Jul 16 12:11:58.916326 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:11:58.916268    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gtqx2" podStartSLOduration=1.916253843 podStartE2EDuration="1.916253843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:11:58.91541878 +0000 UTC m=+1260.216195910" watchObservedRunningTime="2026-07-16 12:11:58.916253843 +0000 UTC m=+1260.217031017"
Jul 16 12:12:09.860215 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:12:09.860175    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gtqx2"
Jul 16 12:18:28.493442 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.493409    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"]
Jul 16 12:18:28.497021 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.497001    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.511227 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.511208    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-59180a3f-openshift-default-dockercfg-zmlhp\""
Jul 16 12:18:28.562800 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.562773    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"]
Jul 16 12:18:28.622609 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.622575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-data\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.622770 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.622619    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-workload-certs\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.622770 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.622654    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-token\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.622770 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.622683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f137f3ab-28d1-43c5-a133-04b0a2290c41-istiod-ca-cert\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.622770 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.622707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-envoy\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.622770 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.622762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-credential-socket\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.622954 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.622793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-podinfo\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.622954 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.622815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvg7g\" (UniqueName: \"kubernetes.io/projected/f137f3ab-28d1-43c5-a133-04b0a2290c41-kube-api-access-tvg7g\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.622954 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.622852    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-workload-socket\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.723663 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.723627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f137f3ab-28d1-43c5-a133-04b0a2290c41-istiod-ca-cert\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.723663 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.723662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-envoy\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.723914 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.723688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-credential-socket\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.723914 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.723809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-podinfo\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.723914 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.723865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvg7g\" (UniqueName: \"kubernetes.io/projected/f137f3ab-28d1-43c5-a133-04b0a2290c41-kube-api-access-tvg7g\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.723914 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.723905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-workload-socket\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.724170 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.724004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-data\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.724170 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.724078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-workload-certs\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.724170 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.724120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-token\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.724609 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.724374    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-workload-socket\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.724609 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.724520    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-data\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.724609 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.724540    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f137f3ab-28d1-43c5-a133-04b0a2290c41-istiod-ca-cert\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.724816 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.724621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-workload-certs\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.724816 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.724706    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-credential-socket\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.726991 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.726963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-envoy\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.727204 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.727184    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-podinfo\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.736236 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.736185    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-token\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.743576 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.743552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvg7g\" (UniqueName: \"kubernetes.io/projected/f137f3ab-28d1-43c5-a133-04b0a2290c41-kube-api-access-tvg7g\") pod \"e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") " pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.806783 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.806759    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:28.971301 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.971267    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"]
Jul 16 12:18:28.974389 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:18:28.974362    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf137f3ab_28d1_43c5_a133_04b0a2290c41.slice/crio-388078f3e601cf63abb56ec8fe0c88d54ebb08aa04b0eee8eb5d5e5a4d30ade9 WatchSource:0}: Error finding container 388078f3e601cf63abb56ec8fe0c88d54ebb08aa04b0eee8eb5d5e5a4d30ade9: Status 404 returned error can't find the container with id 388078f3e601cf63abb56ec8fe0c88d54ebb08aa04b0eee8eb5d5e5a4d30ade9
Jul 16 12:18:28.976454 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.976422    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:18:28.976512 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.976486    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:18:28.976548 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:28.976514    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:18:29.692546 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:29.692502    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk" event={"ID":"f137f3ab-28d1-43c5-a133-04b0a2290c41","Type":"ContainerStarted","Data":"a5e225e4eb4bf8f5b19cf0bbd012617775bcaa9c4e2fcefee929556160b3710b"}
Jul 16 12:18:29.693017 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:29.692574    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk" event={"ID":"f137f3ab-28d1-43c5-a133-04b0a2290c41","Type":"ContainerStarted","Data":"388078f3e601cf63abb56ec8fe0c88d54ebb08aa04b0eee8eb5d5e5a4d30ade9"}
Jul 16 12:18:29.734214 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:29.734162    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk" podStartSLOduration=1.734146327 podStartE2EDuration="1.734146327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:18:29.731032057 +0000 UTC m=+1651.031809152" watchObservedRunningTime="2026-07-16 12:18:29.734146327 +0000 UTC m=+1651.034923424"
Jul 16 12:18:29.807713 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:29.807679    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:30.813459 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:30.813431    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:31.701666 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:31.701628    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:31.702605 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:31.702584    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:47.743267 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:47.743228    2578 patch_prober.go:28] interesting pod/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 16 12:18:47.743678 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:47.743298    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk" podUID="f137f3ab-28d1-43c5-a133-04b0a2290c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 16 12:18:48.786694 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:48.786661    2578 generic.go:356] "Generic (PLEG): container finished" podID="f137f3ab-28d1-43c5-a133-04b0a2290c41" containerID="a5e225e4eb4bf8f5b19cf0bbd012617775bcaa9c4e2fcefee929556160b3710b" exitCode=0
Jul 16 12:18:48.787122 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:48.786741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk" event={"ID":"f137f3ab-28d1-43c5-a133-04b0a2290c41","Type":"ContainerDied","Data":"a5e225e4eb4bf8f5b19cf0bbd012617775bcaa9c4e2fcefee929556160b3710b"}
Jul 16 12:18:48.787179 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:48.787146    2578 scope.go:117] "RemoveContainer" containerID="a5e225e4eb4bf8f5b19cf0bbd012617775bcaa9c4e2fcefee929556160b3710b"
Jul 16 12:18:48.788214 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:48.788188    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:18:48.788307 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:48.788254    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:18:48.788307 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:48.788284    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:18:49.792810 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:49.792773    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk" event={"ID":"f137f3ab-28d1-43c5-a133-04b0a2290c41","Type":"ContainerStarted","Data":"7c5b5a566f911718b33df8107d5744e72dcb9e3b5530ebefd997c3cccbc3c6d1"}
Jul 16 12:18:49.807852 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:49.807828    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:49.808076 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:49.808056    2578 patch_prober.go:28] interesting pod/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 16 12:18:49.808153 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:49.808103    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk" podUID="f137f3ab-28d1-43c5-a133-04b0a2290c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 16 12:18:50.807098 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:50.807070    2578 patch_prober.go:28] interesting pod/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 16 12:18:50.807486 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:50.807121    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk" podUID="f137f3ab-28d1-43c5-a133-04b0a2290c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 16 12:18:51.808104 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:51.808071    2578 patch_prober.go:28] interesting pod/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 16 12:18:51.808474 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:51.808123    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk" podUID="f137f3ab-28d1-43c5-a133-04b0a2290c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 16 12:18:52.807967 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:52.807931    2578 patch_prober.go:28] interesting pod/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 16 12:18:52.808158 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:52.807988    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk" podUID="f137f3ab-28d1-43c5-a133-04b0a2290c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 16 12:18:53.816942 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:53.816911    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:53.817341 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:53.817294    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:18:53.818115 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:18:53.818093    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:20:00.210749 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:00.210709    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"]
Jul 16 12:20:00.211323 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:00.210959    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk" podUID="f137f3ab-28d1-43c5-a133-04b0a2290c41" containerName="istio-proxy" containerID="cri-o://7c5b5a566f911718b33df8107d5744e72dcb9e3b5530ebefd997c3cccbc3c6d1" gracePeriod=30
Jul 16 12:20:00.869723 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:00.869685    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"]
Jul 16 12:20:00.873113 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:00.873088    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:00.878168 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:00.878147    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-953cf885-gw-openshift-default-dockercfg-546fq\""
Jul 16 12:20:00.897936 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:00.897914    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"]
Jul 16 12:20:01.039940 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.039900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-envoy\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.040152 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.039948    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-credential-socket\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.040152 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.039974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-podinfo\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.040152 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.040079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-data\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.040152 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.040144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istiod-ca-cert\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.040317 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.040168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-workload-certs\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.040317 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.040190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8mh7\" (UniqueName: \"kubernetes.io/projected/1282dc1c-07ae-442a-9c07-5ecae1842e3a-kube-api-access-v8mh7\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.040317 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.040213    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-workload-socket\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.040317 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.040287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-token\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.141392 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.141317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istiod-ca-cert\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.141392 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.141352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-workload-certs\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.141392 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.141374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8mh7\" (UniqueName: \"kubernetes.io/projected/1282dc1c-07ae-442a-9c07-5ecae1842e3a-kube-api-access-v8mh7\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.141693 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.141396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-workload-socket\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.141693 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.141430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-token\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.141693 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.141464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-envoy\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.141693 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.141498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-credential-socket\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.141693 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.141525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-podinfo\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.141693 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.141579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-data\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.142103 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.141932    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-workload-socket\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.142103 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.142028    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-workload-certs\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.142103 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.142084    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-data\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.142222 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.142103    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-credential-socket\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.142278 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.142261    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istiod-ca-cert\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.144438 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.144405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-envoy\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.144632 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.144615    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-podinfo\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.152740 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.152719    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8mh7\" (UniqueName: \"kubernetes.io/projected/1282dc1c-07ae-442a-9c07-5ecae1842e3a-kube-api-access-v8mh7\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.153674 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.153652    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-token\") pod \"e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") " pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.183737 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.183712    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:01.333182 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.333160    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"]
Jul 16 12:20:01.335047 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:20:01.335006    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1282dc1c_07ae_442a_9c07_5ecae1842e3a.slice/crio-c19f796bb836306fbe13017a7e1dd68094b52dc7201776c1ab9d406d0bf1d866 WatchSource:0}: Error finding container c19f796bb836306fbe13017a7e1dd68094b52dc7201776c1ab9d406d0bf1d866: Status 404 returned error can't find the container with id c19f796bb836306fbe13017a7e1dd68094b52dc7201776c1ab9d406d0bf1d866
Jul 16 12:20:01.337529 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.337498    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:20:01.337620 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.337562    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:20:01.337620 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:01.337588    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:20:02.142984 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:02.142951    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p" event={"ID":"1282dc1c-07ae-442a-9c07-5ecae1842e3a","Type":"ContainerStarted","Data":"cb520b897df2fd9f6ace23030cdee1f5f39024a1377d126e254314538552cd04"}
Jul 16 12:20:02.143181 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:02.143002    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p" event={"ID":"1282dc1c-07ae-442a-9c07-5ecae1842e3a","Type":"ContainerStarted","Data":"c19f796bb836306fbe13017a7e1dd68094b52dc7201776c1ab9d406d0bf1d866"}
Jul 16 12:20:02.168138 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:02.168089    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p" podStartSLOduration=2.168074632 podStartE2EDuration="2.168074632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:20:02.166271767 +0000 UTC m=+1743.467048885" watchObservedRunningTime="2026-07-16 12:20:02.168074632 +0000 UTC m=+1743.468851730"
Jul 16 12:20:02.184209 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:02.184177    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:02.188848 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:02.188819    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:03.152535 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:03.152486    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:03.153476 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:03.153454    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:05.457976 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.457953    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:20:05.582900 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.582878    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f137f3ab-28d1-43c5-a133-04b0a2290c41-istiod-ca-cert\") pod \"f137f3ab-28d1-43c5-a133-04b0a2290c41\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") "
Jul 16 12:20:05.583026 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.582908    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvg7g\" (UniqueName: \"kubernetes.io/projected/f137f3ab-28d1-43c5-a133-04b0a2290c41-kube-api-access-tvg7g\") pod \"f137f3ab-28d1-43c5-a133-04b0a2290c41\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") "
Jul 16 12:20:05.583026 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.582937    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-envoy\") pod \"f137f3ab-28d1-43c5-a133-04b0a2290c41\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") "
Jul 16 12:20:05.583026 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.582961    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-workload-socket\") pod \"f137f3ab-28d1-43c5-a133-04b0a2290c41\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") "
Jul 16 12:20:05.583026 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.583012    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-podinfo\") pod \"f137f3ab-28d1-43c5-a133-04b0a2290c41\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") "
Jul 16 12:20:05.583266 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.583029    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-data\") pod \"f137f3ab-28d1-43c5-a133-04b0a2290c41\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") "
Jul 16 12:20:05.583330 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.583302    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f137f3ab-28d1-43c5-a133-04b0a2290c41" (UID: "f137f3ab-28d1-43c5-a133-04b0a2290c41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:20:05.583410 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.583385    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-credential-socket\") pod \"f137f3ab-28d1-43c5-a133-04b0a2290c41\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") "
Jul 16 12:20:05.583551 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.583451    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-workload-certs\") pod \"f137f3ab-28d1-43c5-a133-04b0a2290c41\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") "
Jul 16 12:20:05.583551 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.583483    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-token\") pod \"f137f3ab-28d1-43c5-a133-04b0a2290c41\" (UID: \"f137f3ab-28d1-43c5-a133-04b0a2290c41\") "
Jul 16 12:20:05.583672 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.583570    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f137f3ab-28d1-43c5-a133-04b0a2290c41" (UID: "f137f3ab-28d1-43c5-a133-04b0a2290c41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:20:05.583672 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.583591    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f137f3ab-28d1-43c5-a133-04b0a2290c41-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f137f3ab-28d1-43c5-a133-04b0a2290c41" (UID: "f137f3ab-28d1-43c5-a133-04b0a2290c41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 12:20:05.583778 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.583766    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f137f3ab-28d1-43c5-a133-04b0a2290c41" (UID: "f137f3ab-28d1-43c5-a133-04b0a2290c41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:20:05.583926 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.583899    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-data" (OuterVolumeSpecName: "istio-data") pod "f137f3ab-28d1-43c5-a133-04b0a2290c41" (UID: "f137f3ab-28d1-43c5-a133-04b0a2290c41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:20:05.583994 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.583966    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-workload-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:20:05.583994 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.583988    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-credential-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:20:05.584131 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.584005    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-workload-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:20:05.584131 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.584019    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f137f3ab-28d1-43c5-a133-04b0a2290c41-istiod-ca-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:20:05.585184 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.585159    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f137f3ab-28d1-43c5-a133-04b0a2290c41" (UID: "f137f3ab-28d1-43c5-a133-04b0a2290c41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:20:05.585261 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.585194    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f137f3ab-28d1-43c5-a133-04b0a2290c41" (UID: "f137f3ab-28d1-43c5-a133-04b0a2290c41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 12:20:05.585316 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.585296    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f137f3ab-28d1-43c5-a133-04b0a2290c41-kube-api-access-tvg7g" (OuterVolumeSpecName: "kube-api-access-tvg7g") pod "f137f3ab-28d1-43c5-a133-04b0a2290c41" (UID: "f137f3ab-28d1-43c5-a133-04b0a2290c41"). InnerVolumeSpecName "kube-api-access-tvg7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:20:05.585491 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.585476    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-token" (OuterVolumeSpecName: "istio-token") pod "f137f3ab-28d1-43c5-a133-04b0a2290c41" (UID: "f137f3ab-28d1-43c5-a133-04b0a2290c41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:20:05.685113 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.685091    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-envoy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:20:05.685113 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.685115    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-podinfo\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:20:05.685235 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.685124    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-data\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:20:05.685235 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.685133    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f137f3ab-28d1-43c5-a133-04b0a2290c41-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:20:05.685235 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:05.685141    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvg7g\" (UniqueName: \"kubernetes.io/projected/f137f3ab-28d1-43c5-a133-04b0a2290c41-kube-api-access-tvg7g\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:20:06.167936 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:06.167903    2578 generic.go:356] "Generic (PLEG): container finished" podID="f137f3ab-28d1-43c5-a133-04b0a2290c41" containerID="7c5b5a566f911718b33df8107d5744e72dcb9e3b5530ebefd997c3cccbc3c6d1" exitCode=0
Jul 16 12:20:06.168111 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:06.167940    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk" event={"ID":"f137f3ab-28d1-43c5-a133-04b0a2290c41","Type":"ContainerDied","Data":"7c5b5a566f911718b33df8107d5744e72dcb9e3b5530ebefd997c3cccbc3c6d1"}
Jul 16 12:20:06.168111 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:06.168001    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk" event={"ID":"f137f3ab-28d1-43c5-a133-04b0a2290c41","Type":"ContainerDied","Data":"388078f3e601cf63abb56ec8fe0c88d54ebb08aa04b0eee8eb5d5e5a4d30ade9"}
Jul 16 12:20:06.168111 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:06.168022    2578 scope.go:117] "RemoveContainer" containerID="7c5b5a566f911718b33df8107d5744e72dcb9e3b5530ebefd997c3cccbc3c6d1"
Jul 16 12:20:06.168111 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:06.167954    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"
Jul 16 12:20:06.180168 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:06.180146    2578 scope.go:117] "RemoveContainer" containerID="a5e225e4eb4bf8f5b19cf0bbd012617775bcaa9c4e2fcefee929556160b3710b"
Jul 16 12:20:06.188186 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:06.188167    2578 scope.go:117] "RemoveContainer" containerID="7c5b5a566f911718b33df8107d5744e72dcb9e3b5530ebefd997c3cccbc3c6d1"
Jul 16 12:20:06.188470 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:20:06.188448    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5b5a566f911718b33df8107d5744e72dcb9e3b5530ebefd997c3cccbc3c6d1\": container with ID starting with 7c5b5a566f911718b33df8107d5744e72dcb9e3b5530ebefd997c3cccbc3c6d1 not found: ID does not exist" containerID="7c5b5a566f911718b33df8107d5744e72dcb9e3b5530ebefd997c3cccbc3c6d1"
Jul 16 12:20:06.188559 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:06.188476    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5b5a566f911718b33df8107d5744e72dcb9e3b5530ebefd997c3cccbc3c6d1"} err="failed to get container status \"7c5b5a566f911718b33df8107d5744e72dcb9e3b5530ebefd997c3cccbc3c6d1\": rpc error: code = NotFound desc = could not find container \"7c5b5a566f911718b33df8107d5744e72dcb9e3b5530ebefd997c3cccbc3c6d1\": container with ID starting with 7c5b5a566f911718b33df8107d5744e72dcb9e3b5530ebefd997c3cccbc3c6d1 not found: ID does not exist"
Jul 16 12:20:06.188559 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:06.188493    2578 scope.go:117] "RemoveContainer" containerID="a5e225e4eb4bf8f5b19cf0bbd012617775bcaa9c4e2fcefee929556160b3710b"
Jul 16 12:20:06.188715 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:20:06.188699    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e225e4eb4bf8f5b19cf0bbd012617775bcaa9c4e2fcefee929556160b3710b\": container with ID starting with a5e225e4eb4bf8f5b19cf0bbd012617775bcaa9c4e2fcefee929556160b3710b not found: ID does not exist" containerID="a5e225e4eb4bf8f5b19cf0bbd012617775bcaa9c4e2fcefee929556160b3710b"
Jul 16 12:20:06.188751 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:06.188719    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e225e4eb4bf8f5b19cf0bbd012617775bcaa9c4e2fcefee929556160b3710b"} err="failed to get container status \"a5e225e4eb4bf8f5b19cf0bbd012617775bcaa9c4e2fcefee929556160b3710b\": rpc error: code = NotFound desc = could not find container \"a5e225e4eb4bf8f5b19cf0bbd012617775bcaa9c4e2fcefee929556160b3710b\": container with ID starting with a5e225e4eb4bf8f5b19cf0bbd012617775bcaa9c4e2fcefee929556160b3710b not found: ID does not exist"
Jul 16 12:20:06.194974 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:06.194954    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"]
Jul 16 12:20:06.199744 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:06.199723    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-59180a3f-openshift-default-64d7584f6f-cpxvk"]
Jul 16 12:20:07.299643 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:07.299613    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f137f3ab-28d1-43c5-a133-04b0a2290c41" path="/var/lib/kubelet/pods/f137f3ab-28d1-43c5-a133-04b0a2290c41/volumes"
Jul 16 12:20:17.229119 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:17.229081    2578 generic.go:356] "Generic (PLEG): container finished" podID="1282dc1c-07ae-442a-9c07-5ecae1842e3a" containerID="cb520b897df2fd9f6ace23030cdee1f5f39024a1377d126e254314538552cd04" exitCode=0
Jul 16 12:20:17.229514 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:17.229144    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p" event={"ID":"1282dc1c-07ae-442a-9c07-5ecae1842e3a","Type":"ContainerDied","Data":"cb520b897df2fd9f6ace23030cdee1f5f39024a1377d126e254314538552cd04"}
Jul 16 12:20:17.229632 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:17.229614    2578 scope.go:117] "RemoveContainer" containerID="cb520b897df2fd9f6ace23030cdee1f5f39024a1377d126e254314538552cd04"
Jul 16 12:20:17.230804 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:17.230774    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:20:17.230882 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:17.230865    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:20:17.230936 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:17.230920    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:20:18.153161 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:18.153119    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:18.238744 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:18.238702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p" event={"ID":"1282dc1c-07ae-442a-9c07-5ecae1842e3a","Type":"ContainerStarted","Data":"b0b33702f7ca64f3913d0e96cd894b89042facdbacede85545390e821bdd3f78"}
Jul 16 12:20:19.184115 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:19.184071    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:19.185314 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:19.185287    2578 patch_prober.go:28] interesting pod/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 16 12:20:19.185445 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:19.185338    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p" podUID="1282dc1c-07ae-442a-9c07-5ecae1842e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 16 12:20:20.184826 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:20.184789    2578 patch_prober.go:28] interesting pod/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 16 12:20:20.185269 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:20.184852    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p" podUID="1282dc1c-07ae-442a-9c07-5ecae1842e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 16 12:20:21.184967 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:21.184926    2578 patch_prober.go:28] interesting pod/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 16 12:20:21.185408 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:21.185004    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p" podUID="1282dc1c-07ae-442a-9c07-5ecae1842e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 16 12:20:22.184402 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:22.184368    2578 patch_prober.go:28] interesting pod/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 16 12:20:22.184576 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:22.184434    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p" podUID="1282dc1c-07ae-442a-9c07-5ecae1842e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 16 12:20:23.188585 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:23.188553    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:23.188985 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:23.188828    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:20:23.189464 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:20:23.189446    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:21:06.842805 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:06.842770    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"]
Jul 16 12:21:06.843432 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:06.842999    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p" podUID="1282dc1c-07ae-442a-9c07-5ecae1842e3a" containerName="istio-proxy" containerID="cri-o://b0b33702f7ca64f3913d0e96cd894b89042facdbacede85545390e821bdd3f78" gracePeriod=30
Jul 16 12:21:08.191933 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:08.191891    2578 patch_prober.go:28] interesting pod/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 12:21:08.192345 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:08.191950    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p" podUID="1282dc1c-07ae-442a-9c07-5ecae1842e3a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 12:21:12.103976 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.103903    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:21:12.172643 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.172615    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-data\") pod \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") "
Jul 16 12:21:12.172643 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.172646    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8mh7\" (UniqueName: \"kubernetes.io/projected/1282dc1c-07ae-442a-9c07-5ecae1842e3a-kube-api-access-v8mh7\") pod \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") "
Jul 16 12:21:12.172852 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.172688    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-token\") pod \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") "
Jul 16 12:21:12.172852 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.172715    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-workload-certs\") pod \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") "
Jul 16 12:21:12.172852 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.172758    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-podinfo\") pod \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") "
Jul 16 12:21:12.172852 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.172785    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-credential-socket\") pod \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") "
Jul 16 12:21:12.172852 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.172813    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istiod-ca-cert\") pod \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") "
Jul 16 12:21:12.172852 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.172851    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-envoy\") pod \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") "
Jul 16 12:21:12.173270 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.172874    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-workload-socket\") pod \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\" (UID: \"1282dc1c-07ae-442a-9c07-5ecae1842e3a\") "
Jul 16 12:21:12.173270 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.172965    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1282dc1c-07ae-442a-9c07-5ecae1842e3a" (UID: "1282dc1c-07ae-442a-9c07-5ecae1842e3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:21:12.173270 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.173170    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1282dc1c-07ae-442a-9c07-5ecae1842e3a" (UID: "1282dc1c-07ae-442a-9c07-5ecae1842e3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:21:12.173270 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.173251    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1282dc1c-07ae-442a-9c07-5ecae1842e3a" (UID: "1282dc1c-07ae-442a-9c07-5ecae1842e3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 12:21:12.173486 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.173465    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1282dc1c-07ae-442a-9c07-5ecae1842e3a" (UID: "1282dc1c-07ae-442a-9c07-5ecae1842e3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:21:12.173576 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.173555    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-data" (OuterVolumeSpecName: "istio-data") pod "1282dc1c-07ae-442a-9c07-5ecae1842e3a" (UID: "1282dc1c-07ae-442a-9c07-5ecae1842e3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:21:12.173713 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.173579    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-workload-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:21:12.173713 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.173596    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-credential-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:21:12.173713 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.173611    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istiod-ca-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:21:12.173713 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.173623    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-workload-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:21:12.175293 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.175258    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-token" (OuterVolumeSpecName: "istio-token") pod "1282dc1c-07ae-442a-9c07-5ecae1842e3a" (UID: "1282dc1c-07ae-442a-9c07-5ecae1842e3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:21:12.175293 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.175281    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1282dc1c-07ae-442a-9c07-5ecae1842e3a" (UID: "1282dc1c-07ae-442a-9c07-5ecae1842e3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 12:21:12.175433 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.175342    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1282dc1c-07ae-442a-9c07-5ecae1842e3a" (UID: "1282dc1c-07ae-442a-9c07-5ecae1842e3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:21:12.175433 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.175395    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1282dc1c-07ae-442a-9c07-5ecae1842e3a-kube-api-access-v8mh7" (OuterVolumeSpecName: "kube-api-access-v8mh7") pod "1282dc1c-07ae-442a-9c07-5ecae1842e3a" (UID: "1282dc1c-07ae-442a-9c07-5ecae1842e3a"). InnerVolumeSpecName "kube-api-access-v8mh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:21:12.274674 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.274647    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:21:12.274674 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.274672    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-podinfo\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:21:12.274674 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.274680    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-envoy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:21:12.274877 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.274689    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1282dc1c-07ae-442a-9c07-5ecae1842e3a-istio-data\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:21:12.274877 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.274710    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8mh7\" (UniqueName: \"kubernetes.io/projected/1282dc1c-07ae-442a-9c07-5ecae1842e3a-kube-api-access-v8mh7\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:21:12.506102 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.506005    2578 generic.go:356] "Generic (PLEG): container finished" podID="1282dc1c-07ae-442a-9c07-5ecae1842e3a" containerID="b0b33702f7ca64f3913d0e96cd894b89042facdbacede85545390e821bdd3f78" exitCode=0
Jul 16 12:21:12.506102 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.506071    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"
Jul 16 12:21:12.506390 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.506093    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p" event={"ID":"1282dc1c-07ae-442a-9c07-5ecae1842e3a","Type":"ContainerDied","Data":"b0b33702f7ca64f3913d0e96cd894b89042facdbacede85545390e821bdd3f78"}
Jul 16 12:21:12.506390 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.506146    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p" event={"ID":"1282dc1c-07ae-442a-9c07-5ecae1842e3a","Type":"ContainerDied","Data":"c19f796bb836306fbe13017a7e1dd68094b52dc7201776c1ab9d406d0bf1d866"}
Jul 16 12:21:12.506390 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.506167    2578 scope.go:117] "RemoveContainer" containerID="b0b33702f7ca64f3913d0e96cd894b89042facdbacede85545390e821bdd3f78"
Jul 16 12:21:12.515621 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.515605    2578 scope.go:117] "RemoveContainer" containerID="cb520b897df2fd9f6ace23030cdee1f5f39024a1377d126e254314538552cd04"
Jul 16 12:21:12.523074 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.523031    2578 scope.go:117] "RemoveContainer" containerID="b0b33702f7ca64f3913d0e96cd894b89042facdbacede85545390e821bdd3f78"
Jul 16 12:21:12.523394 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:21:12.523368    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0b33702f7ca64f3913d0e96cd894b89042facdbacede85545390e821bdd3f78\": container with ID starting with b0b33702f7ca64f3913d0e96cd894b89042facdbacede85545390e821bdd3f78 not found: ID does not exist" containerID="b0b33702f7ca64f3913d0e96cd894b89042facdbacede85545390e821bdd3f78"
Jul 16 12:21:12.523475 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.523406    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0b33702f7ca64f3913d0e96cd894b89042facdbacede85545390e821bdd3f78"} err="failed to get container status \"b0b33702f7ca64f3913d0e96cd894b89042facdbacede85545390e821bdd3f78\": rpc error: code = NotFound desc = could not find container \"b0b33702f7ca64f3913d0e96cd894b89042facdbacede85545390e821bdd3f78\": container with ID starting with b0b33702f7ca64f3913d0e96cd894b89042facdbacede85545390e821bdd3f78 not found: ID does not exist"
Jul 16 12:21:12.523475 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.523428    2578 scope.go:117] "RemoveContainer" containerID="cb520b897df2fd9f6ace23030cdee1f5f39024a1377d126e254314538552cd04"
Jul 16 12:21:12.523713 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:21:12.523693    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb520b897df2fd9f6ace23030cdee1f5f39024a1377d126e254314538552cd04\": container with ID starting with cb520b897df2fd9f6ace23030cdee1f5f39024a1377d126e254314538552cd04 not found: ID does not exist" containerID="cb520b897df2fd9f6ace23030cdee1f5f39024a1377d126e254314538552cd04"
Jul 16 12:21:12.523779 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.523723    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb520b897df2fd9f6ace23030cdee1f5f39024a1377d126e254314538552cd04"} err="failed to get container status \"cb520b897df2fd9f6ace23030cdee1f5f39024a1377d126e254314538552cd04\": rpc error: code = NotFound desc = could not find container \"cb520b897df2fd9f6ace23030cdee1f5f39024a1377d126e254314538552cd04\": container with ID starting with cb520b897df2fd9f6ace23030cdee1f5f39024a1377d126e254314538552cd04 not found: ID does not exist"
Jul 16 12:21:12.529379 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.529356    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"]
Jul 16 12:21:12.535360 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:12.535338    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-953cf885-gw-openshift-default-67d9855b4c-rgc7p"]
Jul 16 12:21:13.300347 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:13.300315    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1282dc1c-07ae-442a-9c07-5ecae1842e3a" path="/var/lib/kubelet/pods/1282dc1c-07ae-442a-9c07-5ecae1842e3a/volumes"
Jul 16 12:21:19.452745 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.452715    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"]
Jul 16 12:21:19.453113 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.453083    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1282dc1c-07ae-442a-9c07-5ecae1842e3a" containerName="istio-proxy"
Jul 16 12:21:19.453113 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.453096    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1282dc1c-07ae-442a-9c07-5ecae1842e3a" containerName="istio-proxy"
Jul 16 12:21:19.453113 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.453105    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f137f3ab-28d1-43c5-a133-04b0a2290c41" containerName="istio-proxy"
Jul 16 12:21:19.453113 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.453111    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f137f3ab-28d1-43c5-a133-04b0a2290c41" containerName="istio-proxy"
Jul 16 12:21:19.453249 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.453117    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1282dc1c-07ae-442a-9c07-5ecae1842e3a" containerName="istio-proxy"
Jul 16 12:21:19.453249 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.453124    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1282dc1c-07ae-442a-9c07-5ecae1842e3a" containerName="istio-proxy"
Jul 16 12:21:19.453249 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.453188    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f137f3ab-28d1-43c5-a133-04b0a2290c41" containerName="istio-proxy"
Jul 16 12:21:19.453249 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.453196    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f137f3ab-28d1-43c5-a133-04b0a2290c41" containerName="istio-proxy"
Jul 16 12:21:19.453249 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.453204    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1282dc1c-07ae-442a-9c07-5ecae1842e3a" containerName="istio-proxy"
Jul 16 12:21:19.457560 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.457537    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.460205 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.460180    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-05da2a54-openshift-default-dockercfg-rnwj5\""
Jul 16 12:21:19.484212 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.484189    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"]
Jul 16 12:21:19.531108 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.531075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-envoy\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.531233 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.531180    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-token\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.531285 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.531239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istiod-ca-cert\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.531376 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.531351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-credential-socket\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.531419 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.531399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-podinfo\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.531460 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.531436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-workload-socket\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.531504 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.531458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-data\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.531545 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.531505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-workload-certs\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.531545 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.531535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vlb2\" (UniqueName: \"kubernetes.io/projected/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-kube-api-access-8vlb2\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.632242 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.632207    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-credential-socket\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.632442 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.632254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-podinfo\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.632442 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.632290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-workload-socket\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.632442 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.632315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-data\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.632442 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.632342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-workload-certs\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.632442 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.632367    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vlb2\" (UniqueName: \"kubernetes.io/projected/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-kube-api-access-8vlb2\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.632442 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.632406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-envoy\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.632727 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.632467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-token\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.632727 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.632514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istiod-ca-cert\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.632939 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.632884    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-workload-certs\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.632939 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.632905    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-data\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.633160 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.632975    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-workload-socket\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.633471 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.633438    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-credential-socket\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.633747 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.633726    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istiod-ca-cert\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.635208 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.635179    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-envoy\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.635394 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.635372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-podinfo\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.640910 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.640885    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vlb2\" (UniqueName: \"kubernetes.io/projected/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-kube-api-access-8vlb2\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.641626 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.641605    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-token\") pod \"e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") " pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.770095 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.769974    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:19.947583 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.947548    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"]
Jul 16 12:21:19.952543 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:21:19.952508    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e5b611_18cf_4dc5_ac43_5bfe617dcb96.slice/crio-aac6a236f2d7da372a47f9f53dbf3ebc65335603e9d36fae9fe64a1f241902df WatchSource:0}: Error finding container aac6a236f2d7da372a47f9f53dbf3ebc65335603e9d36fae9fe64a1f241902df: Status 404 returned error can't find the container with id aac6a236f2d7da372a47f9f53dbf3ebc65335603e9d36fae9fe64a1f241902df
Jul 16 12:21:19.954588 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.954560    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:21:19.954662 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.954622    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:21:19.954662 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:19.954656    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:21:20.545344 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:20.545301    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll" event={"ID":"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96","Type":"ContainerStarted","Data":"e7156e7f475f4f7f3b276125ff308a6a032258dea4496cc19bd80c645abe9da4"}
Jul 16 12:21:20.545724 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:20.545391    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll" event={"ID":"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96","Type":"ContainerStarted","Data":"aac6a236f2d7da372a47f9f53dbf3ebc65335603e9d36fae9fe64a1f241902df"}
Jul 16 12:21:20.567454 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:20.567408    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll" podStartSLOduration=1.567393418 podStartE2EDuration="1.567393418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:21:20.564827477 +0000 UTC m=+1821.865604575" watchObservedRunningTime="2026-07-16 12:21:20.567393418 +0000 UTC m=+1821.868170515"
Jul 16 12:21:20.770786 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:20.770754    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:21.775094 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:21.775063    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:22.554603 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:22.554572    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:22.555636 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:22.555616    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:37.556346 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:37.556312    2578 patch_prober.go:28] interesting pod/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 16 12:21:37.556745 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:37.556370    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll" podUID="d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 16 12:21:37.632761 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:37.632730    2578 generic.go:356] "Generic (PLEG): container finished" podID="d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" containerID="e7156e7f475f4f7f3b276125ff308a6a032258dea4496cc19bd80c645abe9da4" exitCode=0
Jul 16 12:21:37.633215 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:37.632815    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll" event={"ID":"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96","Type":"ContainerDied","Data":"e7156e7f475f4f7f3b276125ff308a6a032258dea4496cc19bd80c645abe9da4"}
Jul 16 12:21:37.633374 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:37.633334    2578 scope.go:117] "RemoveContainer" containerID="e7156e7f475f4f7f3b276125ff308a6a032258dea4496cc19bd80c645abe9da4"
Jul 16 12:21:37.634517 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:37.634480    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:21:37.634587 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:37.634563    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:21:37.634624 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:37.634607    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:21:38.643067 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:38.643012    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll" event={"ID":"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96","Type":"ContainerStarted","Data":"f687bc74a2fb1b4976186c2dfe1c08e0e87534f226a542c689d9a550f6a4d6d0"}
Jul 16 12:21:38.770919 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:38.770834    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:38.771120 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:38.771090    2578 patch_prober.go:28] interesting pod/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 16 12:21:38.771163 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:38.771148    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll" podUID="d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 16 12:21:39.770876 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:39.770836    2578 patch_prober.go:28] interesting pod/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 16 12:21:39.771271 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:39.770907    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll" podUID="d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 16 12:21:40.770846 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:40.770811    2578 patch_prober.go:28] interesting pod/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 16 12:21:40.771023 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:40.770861    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll" podUID="d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 16 12:21:41.770864 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:41.770832    2578 patch_prober.go:28] interesting pod/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 16 12:21:41.771061 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:41.770881    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll" podUID="d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 16 12:21:42.774084 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:42.774054    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:42.774515 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:42.774503    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:21:42.775092 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:21:42.775074    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:22:51.973346 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:51.973307    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"]
Jul 16 12:22:51.973830 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:51.973502    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll" podUID="d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" containerName="istio-proxy" containerID="cri-o://f687bc74a2fb1b4976186c2dfe1c08e0e87534f226a542c689d9a550f6a4d6d0" gracePeriod=30
Jul 16 12:22:52.309551 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.309518    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"]
Jul 16 12:22:52.309943 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.309927    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f137f3ab-28d1-43c5-a133-04b0a2290c41" containerName="istio-proxy"
Jul 16 12:22:52.309998 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.309945    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f137f3ab-28d1-43c5-a133-04b0a2290c41" containerName="istio-proxy"
Jul 16 12:22:52.310061 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.310020    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1282dc1c-07ae-442a-9c07-5ecae1842e3a" containerName="istio-proxy"
Jul 16 12:22:52.313102 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.313083    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.315744 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.315721    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-26cae25e-openshift-default-dockercfg-q5rzw\""
Jul 16 12:22:52.327198 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.327174    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"]
Jul 16 12:22:52.341029 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.341003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-envoy\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.341136 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.341086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-token\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.341178 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.341144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-podinfo\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.341214 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.341189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f34c5e-97ca-4b84-b282-69f19d8dab22-istiod-ca-cert\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.341267 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.341220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54sfh\" (UniqueName: \"kubernetes.io/projected/35f34c5e-97ca-4b84-b282-69f19d8dab22-kube-api-access-54sfh\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.341329 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.341307    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-data\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.341384 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.341368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-workload-socket\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.341437 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.341394    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-workload-certs\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.341533 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.341473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-credential-socket\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.442392 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.442359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-data\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.442573 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.442406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-workload-socket\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.442573 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.442424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-workload-certs\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.442573 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.442455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-credential-socket\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.442573 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.442477    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-envoy\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.442573 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.442498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-token\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.442840 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.442665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-podinfo\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.442840 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.442707    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f34c5e-97ca-4b84-b282-69f19d8dab22-istiod-ca-cert\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.442840 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.442736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54sfh\" (UniqueName: \"kubernetes.io/projected/35f34c5e-97ca-4b84-b282-69f19d8dab22-kube-api-access-54sfh\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.442840 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.442813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-data\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.443009 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.442844    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-workload-socket\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.443009 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.442925    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-workload-certs\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.443121 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.443032    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-credential-socket\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.443417 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.443400    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f34c5e-97ca-4b84-b282-69f19d8dab22-istiod-ca-cert\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.444910 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.444887    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-envoy\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.445165 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.445146    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-podinfo\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.452534 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.452502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54sfh\" (UniqueName: \"kubernetes.io/projected/35f34c5e-97ca-4b84-b282-69f19d8dab22-kube-api-access-54sfh\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.452661 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.452647    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-token\") pod \"e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") " pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.624218 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.624115    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:52.765217 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.765195    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"]
Jul 16 12:22:52.767506 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:22:52.767477    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f34c5e_97ca_4b84_b282_69f19d8dab22.slice/crio-06df556cb0205d2a096fc106ce2d936685930c69cfefea3b0b20bad9414d9381 WatchSource:0}: Error finding container 06df556cb0205d2a096fc106ce2d936685930c69cfefea3b0b20bad9414d9381: Status 404 returned error can't find the container with id 06df556cb0205d2a096fc106ce2d936685930c69cfefea3b0b20bad9414d9381
Jul 16 12:22:52.769613 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.769575    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:22:52.769681 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.769651    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:22:52.769763 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:52.769679    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:22:53.006650 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:53.006550    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln" event={"ID":"35f34c5e-97ca-4b84-b282-69f19d8dab22","Type":"ContainerStarted","Data":"94a6d7fe3296fe69486472c5c4caf231dbe52dd5cd8f7dc4a1c72888522e77ff"}
Jul 16 12:22:53.006650 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:53.006618    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln" event={"ID":"35f34c5e-97ca-4b84-b282-69f19d8dab22","Type":"ContainerStarted","Data":"06df556cb0205d2a096fc106ce2d936685930c69cfefea3b0b20bad9414d9381"}
Jul 16 12:22:53.049709 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:53.049658    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln" podStartSLOduration=1.0496417 podStartE2EDuration="1.0496417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:22:53.04519898 +0000 UTC m=+1914.345976121" watchObservedRunningTime="2026-07-16 12:22:53.0496417 +0000 UTC m=+1914.350418798"
Jul 16 12:22:53.625199 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:53.625165    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:53.630597 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:53.630575    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:54.011798 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:54.011710    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:54.012630 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:54.012614    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:22:57.234798 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.234742    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:22:57.287872 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.287847    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istiod-ca-cert\") pod \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") "
Jul 16 12:22:57.287982 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.287888    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-podinfo\") pod \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") "
Jul 16 12:22:57.287982 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.287931    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-workload-certs\") pod \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") "
Jul 16 12:22:57.288120 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.287982    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-credential-socket\") pod \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") "
Jul 16 12:22:57.288120 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.288021    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-data\") pod \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") "
Jul 16 12:22:57.288120 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.288079    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-token\") pod \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") "
Jul 16 12:22:57.288282 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.288177    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-workload-socket\") pod \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") "
Jul 16 12:22:57.288282 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.288214    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-envoy\") pod \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") "
Jul 16 12:22:57.288282 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.288239    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vlb2\" (UniqueName: \"kubernetes.io/projected/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-kube-api-access-8vlb2\") pod \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\" (UID: \"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96\") "
Jul 16 12:22:57.288437 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.288279    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" (UID: "d4e5b611-18cf-4dc5-ac43-5bfe617dcb96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 12:22:57.288437 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.288288    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" (UID: "d4e5b611-18cf-4dc5-ac43-5bfe617dcb96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:22:57.288538 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.288525    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istiod-ca-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:22:57.288594 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.288543    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-workload-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:22:57.288594 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.288546    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" (UID: "d4e5b611-18cf-4dc5-ac43-5bfe617dcb96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:22:57.288594 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.288566    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" (UID: "d4e5b611-18cf-4dc5-ac43-5bfe617dcb96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:22:57.289010 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.288986    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-data" (OuterVolumeSpecName: "istio-data") pod "d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" (UID: "d4e5b611-18cf-4dc5-ac43-5bfe617dcb96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:22:57.290236 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.290196    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" (UID: "d4e5b611-18cf-4dc5-ac43-5bfe617dcb96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 12:22:57.290679 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.290654    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-kube-api-access-8vlb2" (OuterVolumeSpecName: "kube-api-access-8vlb2") pod "d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" (UID: "d4e5b611-18cf-4dc5-ac43-5bfe617dcb96"). InnerVolumeSpecName "kube-api-access-8vlb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:22:57.290771 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.290689    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" (UID: "d4e5b611-18cf-4dc5-ac43-5bfe617dcb96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:22:57.290905 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.290886    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-token" (OuterVolumeSpecName: "istio-token") pod "d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" (UID: "d4e5b611-18cf-4dc5-ac43-5bfe617dcb96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:22:57.389766 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.389742    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-data\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:22:57.389766 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.389766    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:22:57.389906 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.389778    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-workload-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:22:57.389906 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.389788    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-envoy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:22:57.389906 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.389796    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vlb2\" (UniqueName: \"kubernetes.io/projected/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-kube-api-access-8vlb2\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:22:57.389906 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.389805    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-istio-podinfo\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:22:57.389906 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:57.389814    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96-credential-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:22:58.036283 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:58.036249    2578 generic.go:356] "Generic (PLEG): container finished" podID="d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" containerID="f687bc74a2fb1b4976186c2dfe1c08e0e87534f226a542c689d9a550f6a4d6d0" exitCode=0
Jul 16 12:22:58.036478 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:58.036308    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"
Jul 16 12:22:58.036478 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:58.036341    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll" event={"ID":"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96","Type":"ContainerDied","Data":"f687bc74a2fb1b4976186c2dfe1c08e0e87534f226a542c689d9a550f6a4d6d0"}
Jul 16 12:22:58.036478 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:58.036408    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll" event={"ID":"d4e5b611-18cf-4dc5-ac43-5bfe617dcb96","Type":"ContainerDied","Data":"aac6a236f2d7da372a47f9f53dbf3ebc65335603e9d36fae9fe64a1f241902df"}
Jul 16 12:22:58.036478 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:58.036439    2578 scope.go:117] "RemoveContainer" containerID="f687bc74a2fb1b4976186c2dfe1c08e0e87534f226a542c689d9a550f6a4d6d0"
Jul 16 12:22:58.045601 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:58.045585    2578 scope.go:117] "RemoveContainer" containerID="e7156e7f475f4f7f3b276125ff308a6a032258dea4496cc19bd80c645abe9da4"
Jul 16 12:22:58.053159 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:58.053140    2578 scope.go:117] "RemoveContainer" containerID="f687bc74a2fb1b4976186c2dfe1c08e0e87534f226a542c689d9a550f6a4d6d0"
Jul 16 12:22:58.053392 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:22:58.053374    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f687bc74a2fb1b4976186c2dfe1c08e0e87534f226a542c689d9a550f6a4d6d0\": container with ID starting with f687bc74a2fb1b4976186c2dfe1c08e0e87534f226a542c689d9a550f6a4d6d0 not found: ID does not exist" containerID="f687bc74a2fb1b4976186c2dfe1c08e0e87534f226a542c689d9a550f6a4d6d0"
Jul 16 12:22:58.053441 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:58.053400    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f687bc74a2fb1b4976186c2dfe1c08e0e87534f226a542c689d9a550f6a4d6d0"} err="failed to get container status \"f687bc74a2fb1b4976186c2dfe1c08e0e87534f226a542c689d9a550f6a4d6d0\": rpc error: code = NotFound desc = could not find container \"f687bc74a2fb1b4976186c2dfe1c08e0e87534f226a542c689d9a550f6a4d6d0\": container with ID starting with f687bc74a2fb1b4976186c2dfe1c08e0e87534f226a542c689d9a550f6a4d6d0 not found: ID does not exist"
Jul 16 12:22:58.053441 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:58.053417    2578 scope.go:117] "RemoveContainer" containerID="e7156e7f475f4f7f3b276125ff308a6a032258dea4496cc19bd80c645abe9da4"
Jul 16 12:22:58.053637 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:22:58.053619    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7156e7f475f4f7f3b276125ff308a6a032258dea4496cc19bd80c645abe9da4\": container with ID starting with e7156e7f475f4f7f3b276125ff308a6a032258dea4496cc19bd80c645abe9da4 not found: ID does not exist" containerID="e7156e7f475f4f7f3b276125ff308a6a032258dea4496cc19bd80c645abe9da4"
Jul 16 12:22:58.053684 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:58.053641    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7156e7f475f4f7f3b276125ff308a6a032258dea4496cc19bd80c645abe9da4"} err="failed to get container status \"e7156e7f475f4f7f3b276125ff308a6a032258dea4496cc19bd80c645abe9da4\": rpc error: code = NotFound desc = could not find container \"e7156e7f475f4f7f3b276125ff308a6a032258dea4496cc19bd80c645abe9da4\": container with ID starting with e7156e7f475f4f7f3b276125ff308a6a032258dea4496cc19bd80c645abe9da4 not found: ID does not exist"
Jul 16 12:22:58.056713 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:58.056694    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"]
Jul 16 12:22:58.063887 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:58.063868    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-05da2a54-openshift-default-96bdcf799-cdfll"]
Jul 16 12:22:59.300018 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:22:59.299990    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" path="/var/lib/kubelet/pods/d4e5b611-18cf-4dc5-ac43-5bfe617dcb96/volumes"
Jul 16 12:23:09.928537 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:09.927711    2578 patch_prober.go:28] interesting pod/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": read tcp 10.133.0.2:39834->10.133.0.46:15021: read: connection reset by peer" start-of-body=
Jul 16 12:23:09.928537 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:09.927774    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln" podUID="35f34c5e-97ca-4b84-b282-69f19d8dab22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": read tcp 10.133.0.2:39834->10.133.0.46:15021: read: connection reset by peer"
Jul 16 12:23:10.096345 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:10.096311    2578 generic.go:356] "Generic (PLEG): container finished" podID="35f34c5e-97ca-4b84-b282-69f19d8dab22" containerID="94a6d7fe3296fe69486472c5c4caf231dbe52dd5cd8f7dc4a1c72888522e77ff" exitCode=0
Jul 16 12:23:10.096481 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:10.096385    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln" event={"ID":"35f34c5e-97ca-4b84-b282-69f19d8dab22","Type":"ContainerDied","Data":"94a6d7fe3296fe69486472c5c4caf231dbe52dd5cd8f7dc4a1c72888522e77ff"}
Jul 16 12:23:10.096781 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:10.096766    2578 scope.go:117] "RemoveContainer" containerID="94a6d7fe3296fe69486472c5c4caf231dbe52dd5cd8f7dc4a1c72888522e77ff"
Jul 16 12:23:10.097735 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:10.097708    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:23:10.097794 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:10.097785    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:23:10.097833 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:10.097817    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:23:11.102319 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:11.102287    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln" event={"ID":"35f34c5e-97ca-4b84-b282-69f19d8dab22","Type":"ContainerStarted","Data":"8fbc02610d1eadd0c43dba8ccb0e31a30a136453727b59e07d6196fff88f8feb"}
Jul 16 12:23:11.625070 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:11.625014    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:23:11.625483 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:11.625456    2578 patch_prober.go:28] interesting pod/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 16 12:23:11.625592 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:11.625513    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln" podUID="35f34c5e-97ca-4b84-b282-69f19d8dab22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 16 12:23:12.625381 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:12.625337    2578 patch_prober.go:28] interesting pod/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 16 12:23:12.625873 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:12.625407    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln" podUID="35f34c5e-97ca-4b84-b282-69f19d8dab22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 16 12:23:13.625429 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:13.625390    2578 patch_prober.go:28] interesting pod/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 16 12:23:13.626009 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:13.625977    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln" podUID="35f34c5e-97ca-4b84-b282-69f19d8dab22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 16 12:23:14.624693 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:14.624656    2578 patch_prober.go:28] interesting pod/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 16 12:23:14.624982 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:14.624955    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln" podUID="35f34c5e-97ca-4b84-b282-69f19d8dab22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 16 12:23:15.628246 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:15.628217    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:23:15.628676 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:15.628576    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:23:15.629359 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:23:15.629343    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:24:13.218234 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:13.218198    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"]
Jul 16 12:24:13.218784 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:13.218396    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln" podUID="35f34c5e-97ca-4b84-b282-69f19d8dab22" containerName="istio-proxy" containerID="cri-o://8fbc02610d1eadd0c43dba8ccb0e31a30a136453727b59e07d6196fff88f8feb" gracePeriod=30
Jul 16 12:24:15.630659 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:15.630619    2578 patch_prober.go:28] interesting pod/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 12:24:15.631080 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:15.630685    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln" podUID="35f34c5e-97ca-4b84-b282-69f19d8dab22" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 12:24:18.413244 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.413203    2578 generic.go:356] "Generic (PLEG): container finished" podID="35f34c5e-97ca-4b84-b282-69f19d8dab22" containerID="8fbc02610d1eadd0c43dba8ccb0e31a30a136453727b59e07d6196fff88f8feb" exitCode=0
Jul 16 12:24:18.413600 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.413239    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln" event={"ID":"35f34c5e-97ca-4b84-b282-69f19d8dab22","Type":"ContainerDied","Data":"8fbc02610d1eadd0c43dba8ccb0e31a30a136453727b59e07d6196fff88f8feb"}
Jul 16 12:24:18.413600 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.413308    2578 scope.go:117] "RemoveContainer" containerID="94a6d7fe3296fe69486472c5c4caf231dbe52dd5cd8f7dc4a1c72888522e77ff"
Jul 16 12:24:18.471602 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.471578    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:24:18.585586 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.585539    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-data\") pod \"35f34c5e-97ca-4b84-b282-69f19d8dab22\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") "
Jul 16 12:24:18.585750 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.585604    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-podinfo\") pod \"35f34c5e-97ca-4b84-b282-69f19d8dab22\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") "
Jul 16 12:24:18.585750 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.585630    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54sfh\" (UniqueName: \"kubernetes.io/projected/35f34c5e-97ca-4b84-b282-69f19d8dab22-kube-api-access-54sfh\") pod \"35f34c5e-97ca-4b84-b282-69f19d8dab22\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") "
Jul 16 12:24:18.585750 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.585659    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f34c5e-97ca-4b84-b282-69f19d8dab22-istiod-ca-cert\") pod \"35f34c5e-97ca-4b84-b282-69f19d8dab22\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") "
Jul 16 12:24:18.585750 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.585691    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-envoy\") pod \"35f34c5e-97ca-4b84-b282-69f19d8dab22\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") "
Jul 16 12:24:18.585985 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.585782    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-token\") pod \"35f34c5e-97ca-4b84-b282-69f19d8dab22\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") "
Jul 16 12:24:18.585985 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.585822    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-workload-socket\") pod \"35f34c5e-97ca-4b84-b282-69f19d8dab22\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") "
Jul 16 12:24:18.585985 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.585860    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-credential-socket\") pod \"35f34c5e-97ca-4b84-b282-69f19d8dab22\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") "
Jul 16 12:24:18.585985 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.585905    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-workload-certs\") pod \"35f34c5e-97ca-4b84-b282-69f19d8dab22\" (UID: \"35f34c5e-97ca-4b84-b282-69f19d8dab22\") "
Jul 16 12:24:18.586231 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.586100    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f34c5e-97ca-4b84-b282-69f19d8dab22-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "35f34c5e-97ca-4b84-b282-69f19d8dab22" (UID: "35f34c5e-97ca-4b84-b282-69f19d8dab22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 12:24:18.586351 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.586294    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f34c5e-97ca-4b84-b282-69f19d8dab22-istiod-ca-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:24:18.586351 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.586294    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "35f34c5e-97ca-4b84-b282-69f19d8dab22" (UID: "35f34c5e-97ca-4b84-b282-69f19d8dab22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:24:18.586351 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.586326    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "35f34c5e-97ca-4b84-b282-69f19d8dab22" (UID: "35f34c5e-97ca-4b84-b282-69f19d8dab22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:24:18.586647 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.586487    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "35f34c5e-97ca-4b84-b282-69f19d8dab22" (UID: "35f34c5e-97ca-4b84-b282-69f19d8dab22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:24:18.586711 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.586656    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-data" (OuterVolumeSpecName: "istio-data") pod "35f34c5e-97ca-4b84-b282-69f19d8dab22" (UID: "35f34c5e-97ca-4b84-b282-69f19d8dab22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:24:18.588019 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.587990    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "35f34c5e-97ca-4b84-b282-69f19d8dab22" (UID: "35f34c5e-97ca-4b84-b282-69f19d8dab22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 12:24:18.588151 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.588073    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-token" (OuterVolumeSpecName: "istio-token") pod "35f34c5e-97ca-4b84-b282-69f19d8dab22" (UID: "35f34c5e-97ca-4b84-b282-69f19d8dab22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:24:18.588330 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.588304    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f34c5e-97ca-4b84-b282-69f19d8dab22-kube-api-access-54sfh" (OuterVolumeSpecName: "kube-api-access-54sfh") pod "35f34c5e-97ca-4b84-b282-69f19d8dab22" (UID: "35f34c5e-97ca-4b84-b282-69f19d8dab22"). InnerVolumeSpecName "kube-api-access-54sfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:24:18.588396 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.588349    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "35f34c5e-97ca-4b84-b282-69f19d8dab22" (UID: "35f34c5e-97ca-4b84-b282-69f19d8dab22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:24:18.687664 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.687634    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-envoy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:24:18.687664 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.687664    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:24:18.687664 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.687674    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-workload-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:24:18.687908 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.687683    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-credential-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:24:18.687908 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.687691    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-workload-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:24:18.687908 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.687701    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-data\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:24:18.687908 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.687708    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f34c5e-97ca-4b84-b282-69f19d8dab22-istio-podinfo\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:24:18.687908 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:18.687716    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54sfh\" (UniqueName: \"kubernetes.io/projected/35f34c5e-97ca-4b84-b282-69f19d8dab22-kube-api-access-54sfh\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:24:19.419382 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:19.419341    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln" event={"ID":"35f34c5e-97ca-4b84-b282-69f19d8dab22","Type":"ContainerDied","Data":"06df556cb0205d2a096fc106ce2d936685930c69cfefea3b0b20bad9414d9381"}
Jul 16 12:24:19.419781 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:19.419399    2578 scope.go:117] "RemoveContainer" containerID="8fbc02610d1eadd0c43dba8ccb0e31a30a136453727b59e07d6196fff88f8feb"
Jul 16 12:24:19.419781 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:19.419356    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"
Jul 16 12:24:19.445995 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:19.445972    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"]
Jul 16 12:24:19.450809 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:19.450786    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-26cae25e-openshift-default-659c9c68c-7fmln"]
Jul 16 12:24:21.299630 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:24:21.299599    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35f34c5e-97ca-4b84-b282-69f19d8dab22" path="/var/lib/kubelet/pods/35f34c5e-97ca-4b84-b282-69f19d8dab22/volumes"
Jul 16 12:25:51.799724 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.799689    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"]
Jul 16 12:25:51.800203 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.800083    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" containerName="istio-proxy"
Jul 16 12:25:51.800203 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.800100    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" containerName="istio-proxy"
Jul 16 12:25:51.800203 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.800109    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35f34c5e-97ca-4b84-b282-69f19d8dab22" containerName="istio-proxy"
Jul 16 12:25:51.800203 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.800116    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="35f34c5e-97ca-4b84-b282-69f19d8dab22" containerName="istio-proxy"
Jul 16 12:25:51.800203 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.800200    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" containerName="istio-proxy"
Jul 16 12:25:51.800393 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.800212    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="35f34c5e-97ca-4b84-b282-69f19d8dab22" containerName="istio-proxy"
Jul 16 12:25:51.803253 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.803233    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.805550 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.805532    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1b4f12d1-openshift-default-dockercfg-fl72p\""
Jul 16 12:25:51.822861 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.822838    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"]
Jul 16 12:25:51.882205 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.882176    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0dd416-b859-4812-89bd-b42640aedda5-istiod-ca-cert\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.882389 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.882234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-credential-socket\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.882389 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.882312    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-workload-certs\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.882389 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.882357    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-istio-envoy\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.882557 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.882422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-workload-socket\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.882557 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.882457    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0dd416-b859-4812-89bd-b42640aedda5-istio-token\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.882633 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.882552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h4lc\" (UniqueName: \"kubernetes.io/projected/ee0dd416-b859-4812-89bd-b42640aedda5-kube-api-access-2h4lc\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.882633 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.882624    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-istio-data\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.882737 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.882651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0dd416-b859-4812-89bd-b42640aedda5-istio-podinfo\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.983485 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.983450    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-istio-data\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.983485 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.983487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0dd416-b859-4812-89bd-b42640aedda5-istio-podinfo\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.983721 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.983506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0dd416-b859-4812-89bd-b42640aedda5-istiod-ca-cert\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.983721 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.983546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-credential-socket\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.983721 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.983572    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-workload-certs\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.983721 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.983591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-istio-envoy\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.983918 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.983764    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-workload-socket\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.983918 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.983809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0dd416-b859-4812-89bd-b42640aedda5-istio-token\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.983918 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.983863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h4lc\" (UniqueName: \"kubernetes.io/projected/ee0dd416-b859-4812-89bd-b42640aedda5-kube-api-access-2h4lc\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.984105 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.983922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-istio-data\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.984105 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.984024    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-credential-socket\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.984105 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.984092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-workload-certs\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.984246 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.984147    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-workload-socket\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.984350 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.984331    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0dd416-b859-4812-89bd-b42640aedda5-istiod-ca-cert\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.985859 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.985836    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-istio-envoy\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.986251 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.986231    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0dd416-b859-4812-89bd-b42640aedda5-istio-podinfo\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.992638 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.992619    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h4lc\" (UniqueName: \"kubernetes.io/projected/ee0dd416-b859-4812-89bd-b42640aedda5-kube-api-access-2h4lc\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:51.992742 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:51.992730    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0dd416-b859-4812-89bd-b42640aedda5-istio-token\") pod \"e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") " pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:52.113683 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:52.113594    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:52.252139 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:52.252112    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"]
Jul 16 12:25:52.254675 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:25:52.254638    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee0dd416_b859_4812_89bd_b42640aedda5.slice/crio-f892834682700bfe1f19c3a42b67b7116a747ad359086f4907c9706da2ddf693 WatchSource:0}: Error finding container f892834682700bfe1f19c3a42b67b7116a747ad359086f4907c9706da2ddf693: Status 404 returned error can't find the container with id f892834682700bfe1f19c3a42b67b7116a747ad359086f4907c9706da2ddf693
Jul 16 12:25:52.256900 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:52.256866    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:25:52.256967 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:52.256934    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:25:52.257007 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:52.256965    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:25:52.858167 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:52.858128    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb" event={"ID":"ee0dd416-b859-4812-89bd-b42640aedda5","Type":"ContainerStarted","Data":"aedbcad868653e87ba3ad140501ebc8d0ff1c03cfc3cad36cfd446f0a4ece151"}
Jul 16 12:25:52.858538 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:52.858185    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb" event={"ID":"ee0dd416-b859-4812-89bd-b42640aedda5","Type":"ContainerStarted","Data":"f892834682700bfe1f19c3a42b67b7116a747ad359086f4907c9706da2ddf693"}
Jul 16 12:25:52.880300 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:52.880232    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb" podStartSLOduration=1.8802180640000001 podStartE2EDuration="1.880218064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:25:52.878202236 +0000 UTC m=+2094.178979334" watchObservedRunningTime="2026-07-16 12:25:52.880218064 +0000 UTC m=+2094.180995161"
Jul 16 12:25:53.114062 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:53.113978    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:53.115423 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:53.115390    2578 patch_prober.go:28] interesting pod/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Jul 16 12:25:53.115545 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:53.115442    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb" podUID="ee0dd416-b859-4812-89bd-b42640aedda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Jul 16 12:25:54.118528 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:54.118498    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:54.867766 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:54.867733    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:25:54.868597 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:25:54.868575    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:26:09.869929 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:09.869893    2578 patch_prober.go:28] interesting pod/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Jul 16 12:26:09.870421 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:09.869953    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb" podUID="ee0dd416-b859-4812-89bd-b42640aedda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Jul 16 12:26:09.939849 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:09.939769    2578 generic.go:356] "Generic (PLEG): container finished" podID="ee0dd416-b859-4812-89bd-b42640aedda5" containerID="aedbcad868653e87ba3ad140501ebc8d0ff1c03cfc3cad36cfd446f0a4ece151" exitCode=0
Jul 16 12:26:09.939849 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:09.939836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb" event={"ID":"ee0dd416-b859-4812-89bd-b42640aedda5","Type":"ContainerDied","Data":"aedbcad868653e87ba3ad140501ebc8d0ff1c03cfc3cad36cfd446f0a4ece151"}
Jul 16 12:26:09.940212 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:09.940199    2578 scope.go:117] "RemoveContainer" containerID="aedbcad868653e87ba3ad140501ebc8d0ff1c03cfc3cad36cfd446f0a4ece151"
Jul 16 12:26:09.941253 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:09.941224    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:26:09.941317 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:09.941292    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:26:09.941359 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:09.941324    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:26:10.949838 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:10.949788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb" event={"ID":"ee0dd416-b859-4812-89bd-b42640aedda5","Type":"ContainerStarted","Data":"a720d6b48f07d43852fdaec35d18987e4c568f45e2b8fe8af60abd09adff8a9a"}
Jul 16 12:26:11.113789 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:11.113748    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:26:11.113992 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:11.113971    2578 patch_prober.go:28] interesting pod/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Jul 16 12:26:11.114075 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:11.114017    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb" podUID="ee0dd416-b859-4812-89bd-b42640aedda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Jul 16 12:26:12.114477 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:12.114437    2578 patch_prober.go:28] interesting pod/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Jul 16 12:26:12.114876 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:12.114521    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb" podUID="ee0dd416-b859-4812-89bd-b42640aedda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Jul 16 12:26:13.114465 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:13.114429    2578 patch_prober.go:28] interesting pod/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Jul 16 12:26:13.114651 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:13.114497    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb" podUID="ee0dd416-b859-4812-89bd-b42640aedda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Jul 16 12:26:14.114430 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:14.114393    2578 patch_prober.go:28] interesting pod/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Jul 16 12:26:14.114621 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:14.114451    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb" podUID="ee0dd416-b859-4812-89bd-b42640aedda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Jul 16 12:26:15.118187 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:15.118153    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:26:15.118575 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:15.118415    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:26:15.119109 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:15.119095    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:26:33.188971 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:33.188940    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"]
Jul 16 12:26:33.189383 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:33.189173    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb" podUID="ee0dd416-b859-4812-89bd-b42640aedda5" containerName="istio-proxy" containerID="cri-o://a720d6b48f07d43852fdaec35d18987e4c568f45e2b8fe8af60abd09adff8a9a" gracePeriod=30
Jul 16 12:26:38.448559 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.448531    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:26:38.577567 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.577542    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0dd416-b859-4812-89bd-b42640aedda5-istiod-ca-cert\") pod \"ee0dd416-b859-4812-89bd-b42640aedda5\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") "
Jul 16 12:26:38.577734 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.577576    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-workload-socket\") pod \"ee0dd416-b859-4812-89bd-b42640aedda5\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") "
Jul 16 12:26:38.577734 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.577596    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0dd416-b859-4812-89bd-b42640aedda5-istio-token\") pod \"ee0dd416-b859-4812-89bd-b42640aedda5\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") "
Jul 16 12:26:38.577734 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.577615    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-istio-envoy\") pod \"ee0dd416-b859-4812-89bd-b42640aedda5\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") "
Jul 16 12:26:38.577734 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.577703    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-credential-socket\") pod \"ee0dd416-b859-4812-89bd-b42640aedda5\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") "
Jul 16 12:26:38.577734 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.577732    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h4lc\" (UniqueName: \"kubernetes.io/projected/ee0dd416-b859-4812-89bd-b42640aedda5-kube-api-access-2h4lc\") pod \"ee0dd416-b859-4812-89bd-b42640aedda5\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") "
Jul 16 12:26:38.578003 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.577772    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0dd416-b859-4812-89bd-b42640aedda5-istio-podinfo\") pod \"ee0dd416-b859-4812-89bd-b42640aedda5\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") "
Jul 16 12:26:38.578003 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.577828    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-istio-data\") pod \"ee0dd416-b859-4812-89bd-b42640aedda5\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") "
Jul 16 12:26:38.578003 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.577864    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-workload-certs\") pod \"ee0dd416-b859-4812-89bd-b42640aedda5\" (UID: \"ee0dd416-b859-4812-89bd-b42640aedda5\") "
Jul 16 12:26:38.578003 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.577896    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee0dd416-b859-4812-89bd-b42640aedda5" (UID: "ee0dd416-b859-4812-89bd-b42640aedda5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:26:38.578003 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.577937    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee0dd416-b859-4812-89bd-b42640aedda5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee0dd416-b859-4812-89bd-b42640aedda5" (UID: "ee0dd416-b859-4812-89bd-b42640aedda5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 12:26:38.578296 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.578058    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee0dd416-b859-4812-89bd-b42640aedda5" (UID: "ee0dd416-b859-4812-89bd-b42640aedda5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:26:38.578296 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.578233    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0dd416-b859-4812-89bd-b42640aedda5-istiod-ca-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:26:38.578296 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.578253    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-workload-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:26:38.578296 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.578252    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee0dd416-b859-4812-89bd-b42640aedda5" (UID: "ee0dd416-b859-4812-89bd-b42640aedda5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:26:38.578296 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.578269    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-credential-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:26:38.578888 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.578858    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee0dd416-b859-4812-89bd-b42640aedda5" (UID: "ee0dd416-b859-4812-89bd-b42640aedda5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:26:38.579728 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.579707    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0dd416-b859-4812-89bd-b42640aedda5-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee0dd416-b859-4812-89bd-b42640aedda5" (UID: "ee0dd416-b859-4812-89bd-b42640aedda5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:26:38.579899 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.579881    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee0dd416-b859-4812-89bd-b42640aedda5" (UID: "ee0dd416-b859-4812-89bd-b42640aedda5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:26:38.580009 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.579985    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0dd416-b859-4812-89bd-b42640aedda5-kube-api-access-2h4lc" (OuterVolumeSpecName: "kube-api-access-2h4lc") pod "ee0dd416-b859-4812-89bd-b42640aedda5" (UID: "ee0dd416-b859-4812-89bd-b42640aedda5"). InnerVolumeSpecName "kube-api-access-2h4lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:26:38.580097 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.580082    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee0dd416-b859-4812-89bd-b42640aedda5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee0dd416-b859-4812-89bd-b42640aedda5" (UID: "ee0dd416-b859-4812-89bd-b42640aedda5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 12:26:38.679654 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.679628    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-workload-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:26:38.679654 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.679652    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0dd416-b859-4812-89bd-b42640aedda5-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:26:38.679793 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.679664    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-istio-envoy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:26:38.679793 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.679676    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h4lc\" (UniqueName: \"kubernetes.io/projected/ee0dd416-b859-4812-89bd-b42640aedda5-kube-api-access-2h4lc\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:26:38.679793 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.679689    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0dd416-b859-4812-89bd-b42640aedda5-istio-podinfo\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:26:38.679793 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:38.679701    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0dd416-b859-4812-89bd-b42640aedda5-istio-data\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:26:39.086014 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:39.085983    2578 generic.go:356] "Generic (PLEG): container finished" podID="ee0dd416-b859-4812-89bd-b42640aedda5" containerID="a720d6b48f07d43852fdaec35d18987e4c568f45e2b8fe8af60abd09adff8a9a" exitCode=0
Jul 16 12:26:39.086205 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:39.086025    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb" event={"ID":"ee0dd416-b859-4812-89bd-b42640aedda5","Type":"ContainerDied","Data":"a720d6b48f07d43852fdaec35d18987e4c568f45e2b8fe8af60abd09adff8a9a"}
Jul 16 12:26:39.086205 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:39.086047    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"
Jul 16 12:26:39.086205 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:39.086093    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb" event={"ID":"ee0dd416-b859-4812-89bd-b42640aedda5","Type":"ContainerDied","Data":"f892834682700bfe1f19c3a42b67b7116a747ad359086f4907c9706da2ddf693"}
Jul 16 12:26:39.086205 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:39.086116    2578 scope.go:117] "RemoveContainer" containerID="a720d6b48f07d43852fdaec35d18987e4c568f45e2b8fe8af60abd09adff8a9a"
Jul 16 12:26:39.098225 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:39.098207    2578 scope.go:117] "RemoveContainer" containerID="aedbcad868653e87ba3ad140501ebc8d0ff1c03cfc3cad36cfd446f0a4ece151"
Jul 16 12:26:39.105829 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:39.105814    2578 scope.go:117] "RemoveContainer" containerID="a720d6b48f07d43852fdaec35d18987e4c568f45e2b8fe8af60abd09adff8a9a"
Jul 16 12:26:39.106075 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:26:39.106058    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a720d6b48f07d43852fdaec35d18987e4c568f45e2b8fe8af60abd09adff8a9a\": container with ID starting with a720d6b48f07d43852fdaec35d18987e4c568f45e2b8fe8af60abd09adff8a9a not found: ID does not exist" containerID="a720d6b48f07d43852fdaec35d18987e4c568f45e2b8fe8af60abd09adff8a9a"
Jul 16 12:26:39.106146 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:39.106083    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a720d6b48f07d43852fdaec35d18987e4c568f45e2b8fe8af60abd09adff8a9a"} err="failed to get container status \"a720d6b48f07d43852fdaec35d18987e4c568f45e2b8fe8af60abd09adff8a9a\": rpc error: code = NotFound desc = could not find container \"a720d6b48f07d43852fdaec35d18987e4c568f45e2b8fe8af60abd09adff8a9a\": container with ID starting with a720d6b48f07d43852fdaec35d18987e4c568f45e2b8fe8af60abd09adff8a9a not found: ID does not exist"
Jul 16 12:26:39.106146 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:39.106099    2578 scope.go:117] "RemoveContainer" containerID="aedbcad868653e87ba3ad140501ebc8d0ff1c03cfc3cad36cfd446f0a4ece151"
Jul 16 12:26:39.106355 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:26:39.106338    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aedbcad868653e87ba3ad140501ebc8d0ff1c03cfc3cad36cfd446f0a4ece151\": container with ID starting with aedbcad868653e87ba3ad140501ebc8d0ff1c03cfc3cad36cfd446f0a4ece151 not found: ID does not exist" containerID="aedbcad868653e87ba3ad140501ebc8d0ff1c03cfc3cad36cfd446f0a4ece151"
Jul 16 12:26:39.106396 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:39.106362    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aedbcad868653e87ba3ad140501ebc8d0ff1c03cfc3cad36cfd446f0a4ece151"} err="failed to get container status \"aedbcad868653e87ba3ad140501ebc8d0ff1c03cfc3cad36cfd446f0a4ece151\": rpc error: code = NotFound desc = could not find container \"aedbcad868653e87ba3ad140501ebc8d0ff1c03cfc3cad36cfd446f0a4ece151\": container with ID starting with aedbcad868653e87ba3ad140501ebc8d0ff1c03cfc3cad36cfd446f0a4ece151 not found: ID does not exist"
Jul 16 12:26:39.113014 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:39.112993    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"]
Jul 16 12:26:39.117465 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:39.117444    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1b4f12d1-openshift-default-64bc9956f9-6c4cb"]
Jul 16 12:26:39.300098 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:39.300066    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee0dd416-b859-4812-89bd-b42640aedda5" path="/var/lib/kubelet/pods/ee0dd416-b859-4812-89bd-b42640aedda5/volumes"
Jul 16 12:26:54.712127 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.712049    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"]
Jul 16 12:26:54.712592 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.712447    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee0dd416-b859-4812-89bd-b42640aedda5" containerName="istio-proxy"
Jul 16 12:26:54.712592 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.712459    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee0dd416-b859-4812-89bd-b42640aedda5" containerName="istio-proxy"
Jul 16 12:26:54.712592 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.712487    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" containerName="istio-proxy"
Jul 16 12:26:54.712592 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.712492    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" containerName="istio-proxy"
Jul 16 12:26:54.712592 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.712498    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35f34c5e-97ca-4b84-b282-69f19d8dab22" containerName="istio-proxy"
Jul 16 12:26:54.712592 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.712504    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="35f34c5e-97ca-4b84-b282-69f19d8dab22" containerName="istio-proxy"
Jul 16 12:26:54.712592 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.712561    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee0dd416-b859-4812-89bd-b42640aedda5" containerName="istio-proxy"
Jul 16 12:26:54.712592 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.712567    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4e5b611-18cf-4dc5-ac43-5bfe617dcb96" containerName="istio-proxy"
Jul 16 12:26:54.712592 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.712574    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="35f34c5e-97ca-4b84-b282-69f19d8dab22" containerName="istio-proxy"
Jul 16 12:26:54.712592 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.712579    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee0dd416-b859-4812-89bd-b42640aedda5" containerName="istio-proxy"
Jul 16 12:26:54.716904 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.716883    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.719688 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.719668    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4a07820d-openshift-default-dockercfg-79nml\""
Jul 16 12:26:54.732772 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.732742    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"]
Jul 16 12:26:54.817910 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.817874    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-workload-socket\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.818118 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.817930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-workload-certs\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.818118 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.817975    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-data\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.818118 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.817998    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-credential-socket\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.818118 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.818086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef34d5b-b273-44aa-8c28-92709b0c3893-istiod-ca-cert\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.818306 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.818135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-envoy\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.818306 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.818165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-token\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.818306 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.818182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-podinfo\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.818306 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.818218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swltw\" (UniqueName: \"kubernetes.io/projected/6ef34d5b-b273-44aa-8c28-92709b0c3893-kube-api-access-swltw\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.919466 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.919434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef34d5b-b273-44aa-8c28-92709b0c3893-istiod-ca-cert\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.919466 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.919472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-envoy\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.919676 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.919488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-token\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.919676 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.919613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-podinfo\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.919747 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.919682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swltw\" (UniqueName: \"kubernetes.io/projected/6ef34d5b-b273-44aa-8c28-92709b0c3893-kube-api-access-swltw\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.919911 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.919860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-workload-socket\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.920115 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.919928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-workload-certs\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.920115 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.919957    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-data\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.920115 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.919984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-credential-socket\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.920304 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.920223    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef34d5b-b273-44aa-8c28-92709b0c3893-istiod-ca-cert\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.920304 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.920252    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-workload-socket\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.920304 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.920292    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-workload-certs\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.920474 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.920333    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-data\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.920474 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.920349    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-credential-socket\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.921837 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.921811    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-envoy\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.922029 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.922010    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-podinfo\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.928247 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.928225    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-token\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:54.928590 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:54.928568    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swltw\" (UniqueName: \"kubernetes.io/projected/6ef34d5b-b273-44aa-8c28-92709b0c3893-kube-api-access-swltw\") pod \"e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") " pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:55.027889 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:55.027805    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:55.176681 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:55.176633    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"]
Jul 16 12:26:55.180502 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:26:55.180474    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef34d5b_b273_44aa_8c28_92709b0c3893.slice/crio-3fcf532e0190b0d70d539d3909622d817561ca26c5e3be0352ffab9bfea058af WatchSource:0}: Error finding container 3fcf532e0190b0d70d539d3909622d817561ca26c5e3be0352ffab9bfea058af: Status 404 returned error can't find the container with id 3fcf532e0190b0d70d539d3909622d817561ca26c5e3be0352ffab9bfea058af
Jul 16 12:26:55.182581 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:55.182530    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:26:55.182659 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:55.182629    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:26:55.182696 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:55.182673    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:26:56.165894 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:56.165859    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq" event={"ID":"6ef34d5b-b273-44aa-8c28-92709b0c3893","Type":"ContainerStarted","Data":"ed2f717a6b45ce590f20974b22dd876c68ff06547b863bbdd0b32cde625d0cd7"}
Jul 16 12:26:56.166303 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:56.165915    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq" event={"ID":"6ef34d5b-b273-44aa-8c28-92709b0c3893","Type":"ContainerStarted","Data":"3fcf532e0190b0d70d539d3909622d817561ca26c5e3be0352ffab9bfea058af"}
Jul 16 12:26:56.190610 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:56.190564    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq" podStartSLOduration=2.190549109 podStartE2EDuration="2.190549109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:26:56.189139908 +0000 UTC m=+2157.489917041" watchObservedRunningTime="2026-07-16 12:26:56.190549109 +0000 UTC m=+2157.491326207"
Jul 16 12:26:57.028383 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:57.028349    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:57.033502 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:57.033470    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:57.173760 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:57.173726    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:26:57.174802 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:26:57.174782    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:27:12.917345 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:12.917030    2578 patch_prober.go:28] interesting pod/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": read tcp 10.133.0.2:32848->10.133.0.48:15021: read: connection reset by peer" start-of-body=
Jul 16 12:27:12.917345 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:12.917213    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq" podUID="6ef34d5b-b273-44aa-8c28-92709b0c3893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": read tcp 10.133.0.2:32848->10.133.0.48:15021: read: connection reset by peer"
Jul 16 12:27:13.260872 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:13.260770    2578 generic.go:356] "Generic (PLEG): container finished" podID="6ef34d5b-b273-44aa-8c28-92709b0c3893" containerID="ed2f717a6b45ce590f20974b22dd876c68ff06547b863bbdd0b32cde625d0cd7" exitCode=0
Jul 16 12:27:13.260872 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:13.260844    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq" event={"ID":"6ef34d5b-b273-44aa-8c28-92709b0c3893","Type":"ContainerDied","Data":"ed2f717a6b45ce590f20974b22dd876c68ff06547b863bbdd0b32cde625d0cd7"}
Jul 16 12:27:13.261268 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:13.261254    2578 scope.go:117] "RemoveContainer" containerID="ed2f717a6b45ce590f20974b22dd876c68ff06547b863bbdd0b32cde625d0cd7"
Jul 16 12:27:13.262307 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:13.262279    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:27:13.262378 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:13.262363    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:27:13.262422 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:13.262404    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:27:14.267675 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:14.267639    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq" event={"ID":"6ef34d5b-b273-44aa-8c28-92709b0c3893","Type":"ContainerStarted","Data":"5953da6f488bcb2401c2aed07bed8ceb9a8c182478f9a107c2aef7cfebfca2ea"}
Jul 16 12:27:15.034442 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:15.034410    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:27:15.034792 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:15.034716    2578 patch_prober.go:28] interesting pod/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 16 12:27:15.034951 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:15.034929    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq" podUID="6ef34d5b-b273-44aa-8c28-92709b0c3893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 16 12:27:16.028518 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:16.028481    2578 patch_prober.go:28] interesting pod/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 16 12:27:16.028998 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:16.028545    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq" podUID="6ef34d5b-b273-44aa-8c28-92709b0c3893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 16 12:27:17.029154 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:17.029106    2578 patch_prober.go:28] interesting pod/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 16 12:27:17.029629 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:17.029179    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq" podUID="6ef34d5b-b273-44aa-8c28-92709b0c3893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 16 12:27:18.028247 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:18.028209    2578 patch_prober.go:28] interesting pod/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 16 12:27:18.028520 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:18.028490    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq" podUID="6ef34d5b-b273-44aa-8c28-92709b0c3893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 16 12:27:19.032178 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:19.032146    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:27:19.032638 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:19.032409    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:27:19.033213 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:27:19.033196    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:28:43.637598 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:43.637563    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"]
Jul 16 12:28:43.638071 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:43.637782    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq" podUID="6ef34d5b-b273-44aa-8c28-92709b0c3893" containerName="istio-proxy" containerID="cri-o://5953da6f488bcb2401c2aed07bed8ceb9a8c182478f9a107c2aef7cfebfca2ea" gracePeriod=30
Jul 16 12:28:48.893220 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:48.893192    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:28:48.940530 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:48.940461    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-envoy\") pod \"6ef34d5b-b273-44aa-8c28-92709b0c3893\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") "
Jul 16 12:28:48.940530 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:48.940503    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-podinfo\") pod \"6ef34d5b-b273-44aa-8c28-92709b0c3893\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") "
Jul 16 12:28:48.940530 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:48.940523    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-workload-certs\") pod \"6ef34d5b-b273-44aa-8c28-92709b0c3893\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") "
Jul 16 12:28:48.940790 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:48.940544    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swltw\" (UniqueName: \"kubernetes.io/projected/6ef34d5b-b273-44aa-8c28-92709b0c3893-kube-api-access-swltw\") pod \"6ef34d5b-b273-44aa-8c28-92709b0c3893\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") "
Jul 16 12:28:48.940790 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:48.940603    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-data\") pod \"6ef34d5b-b273-44aa-8c28-92709b0c3893\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") "
Jul 16 12:28:48.940790 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:48.940649    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-workload-socket\") pod \"6ef34d5b-b273-44aa-8c28-92709b0c3893\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") "
Jul 16 12:28:48.940790 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:48.940684    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef34d5b-b273-44aa-8c28-92709b0c3893-istiod-ca-cert\") pod \"6ef34d5b-b273-44aa-8c28-92709b0c3893\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") "
Jul 16 12:28:48.940790 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:48.940718    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-credential-socket\") pod \"6ef34d5b-b273-44aa-8c28-92709b0c3893\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") "
Jul 16 12:28:48.940790 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:48.940758    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-token\") pod \"6ef34d5b-b273-44aa-8c28-92709b0c3893\" (UID: \"6ef34d5b-b273-44aa-8c28-92709b0c3893\") "
Jul 16 12:28:48.940790 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:48.940780    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6ef34d5b-b273-44aa-8c28-92709b0c3893" (UID: "6ef34d5b-b273-44aa-8c28-92709b0c3893"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:28:48.941165 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:48.941025    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6ef34d5b-b273-44aa-8c28-92709b0c3893" (UID: "6ef34d5b-b273-44aa-8c28-92709b0c3893"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:28:48.941165 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:48.941106    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-workload-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:28:48.941165 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:48.941103    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef34d5b-b273-44aa-8c28-92709b0c3893-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6ef34d5b-b273-44aa-8c28-92709b0c3893" (UID: "6ef34d5b-b273-44aa-8c28-92709b0c3893"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 12:28:48.941333 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:48.941318    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6ef34d5b-b273-44aa-8c28-92709b0c3893" (UID: "6ef34d5b-b273-44aa-8c28-92709b0c3893"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:28:48.941787 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:48.941756    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-data" (OuterVolumeSpecName: "istio-data") pod "6ef34d5b-b273-44aa-8c28-92709b0c3893" (UID: "6ef34d5b-b273-44aa-8c28-92709b0c3893"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:28:48.943215 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:48.943187    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef34d5b-b273-44aa-8c28-92709b0c3893-kube-api-access-swltw" (OuterVolumeSpecName: "kube-api-access-swltw") pod "6ef34d5b-b273-44aa-8c28-92709b0c3893" (UID: "6ef34d5b-b273-44aa-8c28-92709b0c3893"). InnerVolumeSpecName "kube-api-access-swltw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:28:48.943215 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:48.943204    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6ef34d5b-b273-44aa-8c28-92709b0c3893" (UID: "6ef34d5b-b273-44aa-8c28-92709b0c3893"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 12:28:48.943493 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:48.943470    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6ef34d5b-b273-44aa-8c28-92709b0c3893" (UID: "6ef34d5b-b273-44aa-8c28-92709b0c3893"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:28:48.943582 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:48.943488    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-token" (OuterVolumeSpecName: "istio-token") pod "6ef34d5b-b273-44aa-8c28-92709b0c3893" (UID: "6ef34d5b-b273-44aa-8c28-92709b0c3893"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:28:49.041644 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:49.041612    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-envoy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:28:49.041644 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:49.041639    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-podinfo\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:28:49.041644 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:49.041650    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swltw\" (UniqueName: \"kubernetes.io/projected/6ef34d5b-b273-44aa-8c28-92709b0c3893-kube-api-access-swltw\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:28:49.041856 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:49.041660    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-data\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:28:49.041856 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:49.041668    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-workload-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:28:49.041856 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:49.041676    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef34d5b-b273-44aa-8c28-92709b0c3893-istiod-ca-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:28:49.041856 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:49.041685    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef34d5b-b273-44aa-8c28-92709b0c3893-credential-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:28:49.041856 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:49.041692    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef34d5b-b273-44aa-8c28-92709b0c3893-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:28:49.726011 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:49.725919    2578 generic.go:356] "Generic (PLEG): container finished" podID="6ef34d5b-b273-44aa-8c28-92709b0c3893" containerID="5953da6f488bcb2401c2aed07bed8ceb9a8c182478f9a107c2aef7cfebfca2ea" exitCode=0
Jul 16 12:28:49.726205 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:49.726006    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"
Jul 16 12:28:49.726205 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:49.726004    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq" event={"ID":"6ef34d5b-b273-44aa-8c28-92709b0c3893","Type":"ContainerDied","Data":"5953da6f488bcb2401c2aed07bed8ceb9a8c182478f9a107c2aef7cfebfca2ea"}
Jul 16 12:28:49.726205 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:49.726095    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq" event={"ID":"6ef34d5b-b273-44aa-8c28-92709b0c3893","Type":"ContainerDied","Data":"3fcf532e0190b0d70d539d3909622d817561ca26c5e3be0352ffab9bfea058af"}
Jul 16 12:28:49.726205 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:49.726126    2578 scope.go:117] "RemoveContainer" containerID="5953da6f488bcb2401c2aed07bed8ceb9a8c182478f9a107c2aef7cfebfca2ea"
Jul 16 12:28:49.736453 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:49.736431    2578 scope.go:117] "RemoveContainer" containerID="ed2f717a6b45ce590f20974b22dd876c68ff06547b863bbdd0b32cde625d0cd7"
Jul 16 12:28:49.747323 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:49.747282    2578 scope.go:117] "RemoveContainer" containerID="5953da6f488bcb2401c2aed07bed8ceb9a8c182478f9a107c2aef7cfebfca2ea"
Jul 16 12:28:49.747591 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:28:49.747564    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5953da6f488bcb2401c2aed07bed8ceb9a8c182478f9a107c2aef7cfebfca2ea\": container with ID starting with 5953da6f488bcb2401c2aed07bed8ceb9a8c182478f9a107c2aef7cfebfca2ea not found: ID does not exist" containerID="5953da6f488bcb2401c2aed07bed8ceb9a8c182478f9a107c2aef7cfebfca2ea"
Jul 16 12:28:49.747684 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:49.747603    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5953da6f488bcb2401c2aed07bed8ceb9a8c182478f9a107c2aef7cfebfca2ea"} err="failed to get container status \"5953da6f488bcb2401c2aed07bed8ceb9a8c182478f9a107c2aef7cfebfca2ea\": rpc error: code = NotFound desc = could not find container \"5953da6f488bcb2401c2aed07bed8ceb9a8c182478f9a107c2aef7cfebfca2ea\": container with ID starting with 5953da6f488bcb2401c2aed07bed8ceb9a8c182478f9a107c2aef7cfebfca2ea not found: ID does not exist"
Jul 16 12:28:49.747684 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:49.747628    2578 scope.go:117] "RemoveContainer" containerID="ed2f717a6b45ce590f20974b22dd876c68ff06547b863bbdd0b32cde625d0cd7"
Jul 16 12:28:49.747902 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:28:49.747876    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed2f717a6b45ce590f20974b22dd876c68ff06547b863bbdd0b32cde625d0cd7\": container with ID starting with ed2f717a6b45ce590f20974b22dd876c68ff06547b863bbdd0b32cde625d0cd7 not found: ID does not exist" containerID="ed2f717a6b45ce590f20974b22dd876c68ff06547b863bbdd0b32cde625d0cd7"
Jul 16 12:28:49.748024 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:49.747916    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed2f717a6b45ce590f20974b22dd876c68ff06547b863bbdd0b32cde625d0cd7"} err="failed to get container status \"ed2f717a6b45ce590f20974b22dd876c68ff06547b863bbdd0b32cde625d0cd7\": rpc error: code = NotFound desc = could not find container \"ed2f717a6b45ce590f20974b22dd876c68ff06547b863bbdd0b32cde625d0cd7\": container with ID starting with ed2f717a6b45ce590f20974b22dd876c68ff06547b863bbdd0b32cde625d0cd7 not found: ID does not exist"
Jul 16 12:28:49.749961 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:49.749940    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"]
Jul 16 12:28:49.755432 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:49.755407    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4a07820d-openshift-default-649c8bc54c-7fbkq"]
Jul 16 12:28:51.299845 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:28:51.299812    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ef34d5b-b273-44aa-8c28-92709b0c3893" path="/var/lib/kubelet/pods/6ef34d5b-b273-44aa-8c28-92709b0c3893/volumes"
Jul 16 12:29:41.884743 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.884709    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"]
Jul 16 12:29:41.885211 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.885132    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee0dd416-b859-4812-89bd-b42640aedda5" containerName="istio-proxy"
Jul 16 12:29:41.885211 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.885146    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee0dd416-b859-4812-89bd-b42640aedda5" containerName="istio-proxy"
Jul 16 12:29:41.885211 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.885165    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ef34d5b-b273-44aa-8c28-92709b0c3893" containerName="istio-proxy"
Jul 16 12:29:41.885211 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.885170    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ef34d5b-b273-44aa-8c28-92709b0c3893" containerName="istio-proxy"
Jul 16 12:29:41.885211 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.885186    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ef34d5b-b273-44aa-8c28-92709b0c3893" containerName="istio-proxy"
Jul 16 12:29:41.885211 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.885192    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ef34d5b-b273-44aa-8c28-92709b0c3893" containerName="istio-proxy"
Jul 16 12:29:41.885424 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.885254    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ef34d5b-b273-44aa-8c28-92709b0c3893" containerName="istio-proxy"
Jul 16 12:29:41.885424 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.885266    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ef34d5b-b273-44aa-8c28-92709b0c3893" containerName="istio-proxy"
Jul 16 12:29:41.888218 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.888202    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:41.890842 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.890812    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e86929-openshift-default-dockercfg-77dxv\""
Jul 16 12:29:41.918339 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.918308    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"]
Jul 16 12:29:41.989853 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.989820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-workload-socket\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:41.990023 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.989881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-istio-envoy\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:41.990023 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.989900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-istio-data\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:41.990023 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.989916    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2038ace9-23bc-430e-99ee-9d3fed750806-istiod-ca-cert\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:41.990023 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.989954    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-workload-certs\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:41.990023 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.990008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-credential-socket\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:41.990220 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.990078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2038ace9-23bc-430e-99ee-9d3fed750806-istio-token\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:41.990220 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.990107    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcnj5\" (UniqueName: \"kubernetes.io/projected/2038ace9-23bc-430e-99ee-9d3fed750806-kube-api-access-jcnj5\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:41.990220 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:41.990131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2038ace9-23bc-430e-99ee-9d3fed750806-istio-podinfo\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:42.090949 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.090909    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-workload-socket\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:42.091173 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.090969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-istio-envoy\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:42.091173 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.090992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-istio-data\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:42.091173 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.091016    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2038ace9-23bc-430e-99ee-9d3fed750806-istiod-ca-cert\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:42.091173 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.091085    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-workload-certs\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:42.091173 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.091125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-credential-socket\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:42.091173 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.091163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2038ace9-23bc-430e-99ee-9d3fed750806-istio-token\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:42.091494 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.091190    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcnj5\" (UniqueName: \"kubernetes.io/projected/2038ace9-23bc-430e-99ee-9d3fed750806-kube-api-access-jcnj5\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:42.091494 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.091223    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2038ace9-23bc-430e-99ee-9d3fed750806-istio-podinfo\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:42.091494 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.091402    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-workload-socket\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:42.091494 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.091435    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-istio-data\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:42.091860 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.091831    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2038ace9-23bc-430e-99ee-9d3fed750806-istiod-ca-cert\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:42.092005 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.091831    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-workload-certs\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:42.092005 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.091870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-credential-socket\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:42.093352 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.093334    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-istio-envoy\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:42.093795 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.093776    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2038ace9-23bc-430e-99ee-9d3fed750806-istio-podinfo\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:42.104189 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.104170    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2038ace9-23bc-430e-99ee-9d3fed750806-istio-token\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:42.109481 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.109463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcnj5\" (UniqueName: \"kubernetes.io/projected/2038ace9-23bc-430e-99ee-9d3fed750806-kube-api-access-jcnj5\") pod \"e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") " pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:42.199678 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.199596    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:42.344538 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.344511    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"]
Jul 16 12:29:42.346585 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:29:42.346559    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2038ace9_23bc_430e_99ee_9d3fed750806.slice/crio-07f2415a3bbb978baab16f407e8c175cf74360f9ddec07a8ba2a9b4a985fa0cb WatchSource:0}: Error finding container 07f2415a3bbb978baab16f407e8c175cf74360f9ddec07a8ba2a9b4a985fa0cb: Status 404 returned error can't find the container with id 07f2415a3bbb978baab16f407e8c175cf74360f9ddec07a8ba2a9b4a985fa0cb
Jul 16 12:29:42.348682 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.348648    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:29:42.348782 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.348723    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:29:42.348782 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.348766    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:29:42.983443 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.983409    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2" event={"ID":"2038ace9-23bc-430e-99ee-9d3fed750806","Type":"ContainerStarted","Data":"1630e62c7754f5dc288626eb75c1afc63634f5c6be8d180f76e73976166b8b29"}
Jul 16 12:29:42.983854 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:42.983489    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2" event={"ID":"2038ace9-23bc-430e-99ee-9d3fed750806","Type":"ContainerStarted","Data":"07f2415a3bbb978baab16f407e8c175cf74360f9ddec07a8ba2a9b4a985fa0cb"}
Jul 16 12:29:43.007905 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:43.007855    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2" podStartSLOduration=2.00784096 podStartE2EDuration="2.00784096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:29:43.006670289 +0000 UTC m=+2324.307447423" watchObservedRunningTime="2026-07-16 12:29:43.00784096 +0000 UTC m=+2324.308618058"
Jul 16 12:29:43.200354 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:43.200313    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:43.205237 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:43.205209    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:43.989390 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:43.989354    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:43.990393 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:43.990372    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:29:50.357939 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.357898    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"]
Jul 16 12:29:50.361690 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.361669    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.364756 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.364735    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e41619-openshift-default-dockercfg-2knnz\""
Jul 16 12:29:50.382954 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.382932    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"]
Jul 16 12:29:50.467142 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.467095    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-envoy\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.467425 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.467394    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-workload-certs\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.467550 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.467438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-data\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.467550 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.467460    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baaf5878-af82-4b4f-85f4-157d901e8b22-istiod-ca-cert\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.467550 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.467503    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-podinfo\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.467550 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.467540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-credential-socket\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.467730 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.467580    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-token\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.467730 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.467605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr42d\" (UniqueName: \"kubernetes.io/projected/baaf5878-af82-4b4f-85f4-157d901e8b22-kube-api-access-wr42d\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.467730 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.467665    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-workload-socket\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.568727 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.568674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-token\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.568727 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.568730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr42d\" (UniqueName: \"kubernetes.io/projected/baaf5878-af82-4b4f-85f4-157d901e8b22-kube-api-access-wr42d\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.568967 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.568767    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-workload-socket\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.568967 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.568856    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-envoy\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.568967 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.568895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-workload-certs\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.568967 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.568922    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-data\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.568967 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.568947    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baaf5878-af82-4b4f-85f4-157d901e8b22-istiod-ca-cert\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.569267 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.569056    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-podinfo\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.569267 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.569100    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-credential-socket\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.569383 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.569284    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-workload-socket\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.569439 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.569413    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-workload-certs\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.569491 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.569460    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-credential-socket\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.569541 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.569500    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-data\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.569797 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.569778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baaf5878-af82-4b4f-85f4-157d901e8b22-istiod-ca-cert\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.571265 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.571247    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-envoy\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.571465 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.571448    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-podinfo\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.581542 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.581520    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-token\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.587083 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.587052    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr42d\" (UniqueName: \"kubernetes.io/projected/baaf5878-af82-4b4f-85f4-157d901e8b22-kube-api-access-wr42d\") pod \"e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") " pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.672934 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.672859    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:50.828525 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.828489    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"]
Jul 16 12:29:50.834397 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:29:50.834368    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaaf5878_af82_4b4f_85f4_157d901e8b22.slice/crio-4bd275d23b55f9d08572eac928d5ab7b090d7914849bbd68c8f94e3a37c1564b WatchSource:0}: Error finding container 4bd275d23b55f9d08572eac928d5ab7b090d7914849bbd68c8f94e3a37c1564b: Status 404 returned error can't find the container with id 4bd275d23b55f9d08572eac928d5ab7b090d7914849bbd68c8f94e3a37c1564b
Jul 16 12:29:50.836520 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.836487    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:29:50.836604 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.836556    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:29:50.836604 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:50.836583    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:29:51.027830 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:51.027710    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm" event={"ID":"baaf5878-af82-4b4f-85f4-157d901e8b22","Type":"ContainerStarted","Data":"11fedbc4058804413344b9d7f5568ef5e95fb081d40d8158041f723fdf99f75b"}
Jul 16 12:29:51.027830 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:51.027775    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm" event={"ID":"baaf5878-af82-4b4f-85f4-157d901e8b22","Type":"ContainerStarted","Data":"4bd275d23b55f9d08572eac928d5ab7b090d7914849bbd68c8f94e3a37c1564b"}
Jul 16 12:29:51.071214 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:51.071160    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm" podStartSLOduration=1.071139354 podStartE2EDuration="1.071139354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:29:51.069169856 +0000 UTC m=+2332.369946969" watchObservedRunningTime="2026-07-16 12:29:51.071139354 +0000 UTC m=+2332.371916451"
Jul 16 12:29:51.673641 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:51.673599    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:51.678750 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:51.678726    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:52.032702 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:52.032615    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:29:52.033555 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:29:52.033536    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:30:00.030209 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:00.030163    2578 patch_prober.go:28] interesting pod/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 16 12:30:00.030697 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:00.030243    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2" podUID="2038ace9-23bc-430e-99ee-9d3fed750806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 16 12:30:02.088888 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:02.088856    2578 generic.go:356] "Generic (PLEG): container finished" podID="2038ace9-23bc-430e-99ee-9d3fed750806" containerID="1630e62c7754f5dc288626eb75c1afc63634f5c6be8d180f76e73976166b8b29" exitCode=0
Jul 16 12:30:02.089342 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:02.088939    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2" event={"ID":"2038ace9-23bc-430e-99ee-9d3fed750806","Type":"ContainerDied","Data":"1630e62c7754f5dc288626eb75c1afc63634f5c6be8d180f76e73976166b8b29"}
Jul 16 12:30:02.089414 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:02.089387    2578 scope.go:117] "RemoveContainer" containerID="1630e62c7754f5dc288626eb75c1afc63634f5c6be8d180f76e73976166b8b29"
Jul 16 12:30:02.090381 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:02.090355    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:30:02.090443 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:02.090421    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:30:02.090487 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:02.090453    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:30:03.096218 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:03.096180    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2" event={"ID":"2038ace9-23bc-430e-99ee-9d3fed750806","Type":"ContainerStarted","Data":"495dc92682b8125dfc6f0a0df8aebe7353fb2bcd72d7c9fed7248c25edf457b9"}
Jul 16 12:30:03.199908 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:03.199862    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:30:03.200151 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:03.200126    2578 patch_prober.go:28] interesting pod/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 16 12:30:03.200215 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:03.200193    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2" podUID="2038ace9-23bc-430e-99ee-9d3fed750806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 16 12:30:04.200634 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:04.200605    2578 patch_prober.go:28] interesting pod/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 16 12:30:04.200995 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:04.200670    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2" podUID="2038ace9-23bc-430e-99ee-9d3fed750806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 16 12:30:05.200356 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:05.200310    2578 patch_prober.go:28] interesting pod/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 16 12:30:05.200547 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:05.200389    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2" podUID="2038ace9-23bc-430e-99ee-9d3fed750806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 16 12:30:06.200200 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:06.200159    2578 patch_prober.go:28] interesting pod/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 16 12:30:06.200605 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:06.200233    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2" podUID="2038ace9-23bc-430e-99ee-9d3fed750806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 16 12:30:07.200550 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:07.200515    2578 patch_prober.go:28] interesting pod/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 16 12:30:07.201119 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:07.200566    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2" podUID="2038ace9-23bc-430e-99ee-9d3fed750806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 16 12:30:08.205073 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:08.205026    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:30:08.205440 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:08.205371    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:30:08.206058 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:08.206020    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:30:11.137366 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:11.137331    2578 generic.go:356] "Generic (PLEG): container finished" podID="baaf5878-af82-4b4f-85f4-157d901e8b22" containerID="11fedbc4058804413344b9d7f5568ef5e95fb081d40d8158041f723fdf99f75b" exitCode=0
Jul 16 12:30:11.137893 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:11.137411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm" event={"ID":"baaf5878-af82-4b4f-85f4-157d901e8b22","Type":"ContainerDied","Data":"11fedbc4058804413344b9d7f5568ef5e95fb081d40d8158041f723fdf99f75b"}
Jul 16 12:30:11.137975 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:11.137956    2578 scope.go:117] "RemoveContainer" containerID="11fedbc4058804413344b9d7f5568ef5e95fb081d40d8158041f723fdf99f75b"
Jul 16 12:30:11.139000 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:11.138972    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:30:11.139111 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:11.139051    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:30:11.139111 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:11.139084    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:30:12.148282 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:12.148235    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm" event={"ID":"baaf5878-af82-4b4f-85f4-157d901e8b22","Type":"ContainerStarted","Data":"acf16d00349ffb254b6e236240bfd2e726c6dc2d844e78d714a7101be486ecb8"}
Jul 16 12:30:12.673298 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:12.673264    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:30:12.673512 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:12.673484    2578 patch_prober.go:28] interesting pod/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Jul 16 12:30:12.673569 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:12.673546    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm" podUID="baaf5878-af82-4b4f-85f4-157d901e8b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Jul 16 12:30:13.673888 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:13.673855    2578 patch_prober.go:28] interesting pod/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Jul 16 12:30:13.674307 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:13.673908    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm" podUID="baaf5878-af82-4b4f-85f4-157d901e8b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Jul 16 12:30:14.673424 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:14.673387    2578 patch_prober.go:28] interesting pod/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Jul 16 12:30:14.673651 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:14.673441    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm" podUID="baaf5878-af82-4b4f-85f4-157d901e8b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Jul 16 12:30:15.673585 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:15.673546    2578 patch_prober.go:28] interesting pod/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Jul 16 12:30:15.674077 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:15.673605    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm" podUID="baaf5878-af82-4b4f-85f4-157d901e8b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Jul 16 12:30:16.676632 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:16.676601    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:30:16.677089 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:16.676875    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:30:16.677719 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:30:16.677703    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:31:29.071889 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:29.071810    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"]
Jul 16 12:31:29.072410 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:29.072081    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2" podUID="2038ace9-23bc-430e-99ee-9d3fed750806" containerName="istio-proxy" containerID="cri-o://495dc92682b8125dfc6f0a0df8aebe7353fb2bcd72d7c9fed7248c25edf457b9" gracePeriod=30
Jul 16 12:31:34.332585 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.332521    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:31:34.433769 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.433742    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-istio-envoy\") pod \"2038ace9-23bc-430e-99ee-9d3fed750806\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") "
Jul 16 12:31:34.433950 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.433782    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-credential-socket\") pod \"2038ace9-23bc-430e-99ee-9d3fed750806\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") "
Jul 16 12:31:34.433950 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.433840    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2038ace9-23bc-430e-99ee-9d3fed750806-istiod-ca-cert\") pod \"2038ace9-23bc-430e-99ee-9d3fed750806\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") "
Jul 16 12:31:34.433950 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.433863    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcnj5\" (UniqueName: \"kubernetes.io/projected/2038ace9-23bc-430e-99ee-9d3fed750806-kube-api-access-jcnj5\") pod \"2038ace9-23bc-430e-99ee-9d3fed750806\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") "
Jul 16 12:31:34.433950 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.433897    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-workload-socket\") pod \"2038ace9-23bc-430e-99ee-9d3fed750806\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") "
Jul 16 12:31:34.433950 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.433918    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-workload-certs\") pod \"2038ace9-23bc-430e-99ee-9d3fed750806\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") "
Jul 16 12:31:34.434252 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.434062    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-istio-data\") pod \"2038ace9-23bc-430e-99ee-9d3fed750806\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") "
Jul 16 12:31:34.434252 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.434102    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2038ace9-23bc-430e-99ee-9d3fed750806-istio-podinfo\") pod \"2038ace9-23bc-430e-99ee-9d3fed750806\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") "
Jul 16 12:31:34.434252 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.434126    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2038ace9-23bc-430e-99ee-9d3fed750806-istio-token\") pod \"2038ace9-23bc-430e-99ee-9d3fed750806\" (UID: \"2038ace9-23bc-430e-99ee-9d3fed750806\") "
Jul 16 12:31:34.434252 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.434196    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2038ace9-23bc-430e-99ee-9d3fed750806" (UID: "2038ace9-23bc-430e-99ee-9d3fed750806"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:31:34.434458 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.434311    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2038ace9-23bc-430e-99ee-9d3fed750806-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2038ace9-23bc-430e-99ee-9d3fed750806" (UID: "2038ace9-23bc-430e-99ee-9d3fed750806"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 12:31:34.434458 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.434433    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2038ace9-23bc-430e-99ee-9d3fed750806" (UID: "2038ace9-23bc-430e-99ee-9d3fed750806"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:31:34.434768 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.434736    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2038ace9-23bc-430e-99ee-9d3fed750806" (UID: "2038ace9-23bc-430e-99ee-9d3fed750806"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:31:34.435072 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.435003    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-credential-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:31:34.435072 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.435029    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2038ace9-23bc-430e-99ee-9d3fed750806-istiod-ca-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:31:34.435072 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.435063    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-workload-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:31:34.435072 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.435077    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-workload-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:31:34.435548 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.435511    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-istio-data" (OuterVolumeSpecName: "istio-data") pod "2038ace9-23bc-430e-99ee-9d3fed750806" (UID: "2038ace9-23bc-430e-99ee-9d3fed750806"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:31:34.436224 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.436202    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2038ace9-23bc-430e-99ee-9d3fed750806" (UID: "2038ace9-23bc-430e-99ee-9d3fed750806"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:31:34.436727 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.436706    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2038ace9-23bc-430e-99ee-9d3fed750806-istio-token" (OuterVolumeSpecName: "istio-token") pod "2038ace9-23bc-430e-99ee-9d3fed750806" (UID: "2038ace9-23bc-430e-99ee-9d3fed750806"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:31:34.436727 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.436714    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2038ace9-23bc-430e-99ee-9d3fed750806-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2038ace9-23bc-430e-99ee-9d3fed750806" (UID: "2038ace9-23bc-430e-99ee-9d3fed750806"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 12:31:34.436879 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.436857    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2038ace9-23bc-430e-99ee-9d3fed750806-kube-api-access-jcnj5" (OuterVolumeSpecName: "kube-api-access-jcnj5") pod "2038ace9-23bc-430e-99ee-9d3fed750806" (UID: "2038ace9-23bc-430e-99ee-9d3fed750806"). InnerVolumeSpecName "kube-api-access-jcnj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:31:34.536183 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.536150    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcnj5\" (UniqueName: \"kubernetes.io/projected/2038ace9-23bc-430e-99ee-9d3fed750806-kube-api-access-jcnj5\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:31:34.536183 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.536177    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-istio-data\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:31:34.536183 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.536186    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2038ace9-23bc-430e-99ee-9d3fed750806-istio-podinfo\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:31:34.536487 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.536194    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2038ace9-23bc-430e-99ee-9d3fed750806-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:31:34.536487 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.536203    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2038ace9-23bc-430e-99ee-9d3fed750806-istio-envoy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:31:34.540605 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.540576    2578 generic.go:356] "Generic (PLEG): container finished" podID="2038ace9-23bc-430e-99ee-9d3fed750806" containerID="495dc92682b8125dfc6f0a0df8aebe7353fb2bcd72d7c9fed7248c25edf457b9" exitCode=0
Jul 16 12:31:34.540777 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.540626    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"
Jul 16 12:31:34.540777 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.540617    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2" event={"ID":"2038ace9-23bc-430e-99ee-9d3fed750806","Type":"ContainerDied","Data":"495dc92682b8125dfc6f0a0df8aebe7353fb2bcd72d7c9fed7248c25edf457b9"}
Jul 16 12:31:34.540777 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.540763    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2" event={"ID":"2038ace9-23bc-430e-99ee-9d3fed750806","Type":"ContainerDied","Data":"07f2415a3bbb978baab16f407e8c175cf74360f9ddec07a8ba2a9b4a985fa0cb"}
Jul 16 12:31:34.540938 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.540790    2578 scope.go:117] "RemoveContainer" containerID="495dc92682b8125dfc6f0a0df8aebe7353fb2bcd72d7c9fed7248c25edf457b9"
Jul 16 12:31:34.550845 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.550822    2578 scope.go:117] "RemoveContainer" containerID="1630e62c7754f5dc288626eb75c1afc63634f5c6be8d180f76e73976166b8b29"
Jul 16 12:31:34.558654 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.558635    2578 scope.go:117] "RemoveContainer" containerID="495dc92682b8125dfc6f0a0df8aebe7353fb2bcd72d7c9fed7248c25edf457b9"
Jul 16 12:31:34.558907 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:31:34.558888    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"495dc92682b8125dfc6f0a0df8aebe7353fb2bcd72d7c9fed7248c25edf457b9\": container with ID starting with 495dc92682b8125dfc6f0a0df8aebe7353fb2bcd72d7c9fed7248c25edf457b9 not found: ID does not exist" containerID="495dc92682b8125dfc6f0a0df8aebe7353fb2bcd72d7c9fed7248c25edf457b9"
Jul 16 12:31:34.558971 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.558915    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495dc92682b8125dfc6f0a0df8aebe7353fb2bcd72d7c9fed7248c25edf457b9"} err="failed to get container status \"495dc92682b8125dfc6f0a0df8aebe7353fb2bcd72d7c9fed7248c25edf457b9\": rpc error: code = NotFound desc = could not find container \"495dc92682b8125dfc6f0a0df8aebe7353fb2bcd72d7c9fed7248c25edf457b9\": container with ID starting with 495dc92682b8125dfc6f0a0df8aebe7353fb2bcd72d7c9fed7248c25edf457b9 not found: ID does not exist"
Jul 16 12:31:34.558971 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.558931    2578 scope.go:117] "RemoveContainer" containerID="1630e62c7754f5dc288626eb75c1afc63634f5c6be8d180f76e73976166b8b29"
Jul 16 12:31:34.559196 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:31:34.559179    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1630e62c7754f5dc288626eb75c1afc63634f5c6be8d180f76e73976166b8b29\": container with ID starting with 1630e62c7754f5dc288626eb75c1afc63634f5c6be8d180f76e73976166b8b29 not found: ID does not exist" containerID="1630e62c7754f5dc288626eb75c1afc63634f5c6be8d180f76e73976166b8b29"
Jul 16 12:31:34.559243 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.559202    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1630e62c7754f5dc288626eb75c1afc63634f5c6be8d180f76e73976166b8b29"} err="failed to get container status \"1630e62c7754f5dc288626eb75c1afc63634f5c6be8d180f76e73976166b8b29\": rpc error: code = NotFound desc = could not find container \"1630e62c7754f5dc288626eb75c1afc63634f5c6be8d180f76e73976166b8b29\": container with ID starting with 1630e62c7754f5dc288626eb75c1afc63634f5c6be8d180f76e73976166b8b29 not found: ID does not exist"
Jul 16 12:31:34.567684 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.567663    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"]
Jul 16 12:31:34.572103 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:34.572083    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e86929-openshift-default-5fd55d5bb7-hswj2"]
Jul 16 12:31:35.301239 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:35.301211    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2038ace9-23bc-430e-99ee-9d3fed750806" path="/var/lib/kubelet/pods/2038ace9-23bc-430e-99ee-9d3fed750806/volumes"
Jul 16 12:31:55.388991 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:55.388958    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"]
Jul 16 12:31:55.389353 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:55.389190    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm" podUID="baaf5878-af82-4b4f-85f4-157d901e8b22" containerName="istio-proxy" containerID="cri-o://acf16d00349ffb254b6e236240bfd2e726c6dc2d844e78d714a7101be486ecb8" gracePeriod=30
Jul 16 12:31:56.438669 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.438638    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"]
Jul 16 12:31:56.439066 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.439051    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2038ace9-23bc-430e-99ee-9d3fed750806" containerName="istio-proxy"
Jul 16 12:31:56.439066 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.439067    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2038ace9-23bc-430e-99ee-9d3fed750806" containerName="istio-proxy"
Jul 16 12:31:56.439155 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.439079    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2038ace9-23bc-430e-99ee-9d3fed750806" containerName="istio-proxy"
Jul 16 12:31:56.439155 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.439087    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2038ace9-23bc-430e-99ee-9d3fed750806" containerName="istio-proxy"
Jul 16 12:31:56.439236 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.439168    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2038ace9-23bc-430e-99ee-9d3fed750806" containerName="istio-proxy"
Jul 16 12:31:56.439236 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.439181    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2038ace9-23bc-430e-99ee-9d3fed750806" containerName="istio-proxy"
Jul 16 12:31:56.442316 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.442302    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.445996 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:31:56.445965    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" is forbidden: User \"system:node:ip-10-0-129-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-129-163.ec2.internal' and this object" podUID="448a9f63-dbbb-4023-ba86-2760e97e5671" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.446237 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:31:56.446214    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-api-a-2f272c-openshift-default-dockercfg-54p7d\" is forbidden: User \"system:node:ip-10-0-129-163.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-129-163.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2f272c-openshift-default-dockercfg-54p7d\"" type="*v1.Secret"
Jul 16 12:31:56.492768 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.492733    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"]
Jul 16 12:31:56.527525 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.527484    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-workload-certs\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.527686 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.527544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/448a9f63-dbbb-4023-ba86-2760e97e5671-istiod-ca-cert\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.527686 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.527583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-token\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.527686 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.527631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvfnb\" (UniqueName: \"kubernetes.io/projected/448a9f63-dbbb-4023-ba86-2760e97e5671-kube-api-access-nvfnb\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.527686 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.527657    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-credential-socket\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.527868 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.527700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-podinfo\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.527868 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.527764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-workload-socket\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.527868 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.527798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-envoy\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.527868 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.527812    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-data\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.628274 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.628245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-workload-certs\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.628463 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.628284    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/448a9f63-dbbb-4023-ba86-2760e97e5671-istiod-ca-cert\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.628463 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.628302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-token\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.628463 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.628338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvfnb\" (UniqueName: \"kubernetes.io/projected/448a9f63-dbbb-4023-ba86-2760e97e5671-kube-api-access-nvfnb\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.628463 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.628362    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-credential-socket\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.628463 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.628383    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-podinfo\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.628463 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.628439    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-workload-socket\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.628463 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.628465    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-envoy\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.628814 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.628488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-data\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.628814 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.628719    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-workload-certs\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.629126 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.629098    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-data\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.629315 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.629098    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/448a9f63-dbbb-4023-ba86-2760e97e5671-istiod-ca-cert\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.629315 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.629130    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-credential-socket\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.629496 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.629182    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-workload-socket\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.631067 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.631026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-podinfo\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.631166 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.631139    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-envoy\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.648943 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.648920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvfnb\" (UniqueName: \"kubernetes.io/projected/448a9f63-dbbb-4023-ba86-2760e97e5671-kube-api-access-nvfnb\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:56.651466 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:56.651445    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-token\") pod \"e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") " pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:57.257389 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:57.257359    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2f272c-openshift-default-dockercfg-54p7d\""
Jul 16 12:31:57.263414 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:57.263394    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:57.414117 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:57.414084    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"]
Jul 16 12:31:57.419058 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:31:57.419014    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448a9f63_dbbb_4023_ba86_2760e97e5671.slice/crio-8b7f9c333c207cccce3e1552f5804fb1f7aed8f3410c8bf8d59fcf1459b07f64 WatchSource:0}: Error finding container 8b7f9c333c207cccce3e1552f5804fb1f7aed8f3410c8bf8d59fcf1459b07f64: Status 404 returned error can't find the container with id 8b7f9c333c207cccce3e1552f5804fb1f7aed8f3410c8bf8d59fcf1459b07f64
Jul 16 12:31:57.421190 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:57.421152    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:31:57.421271 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:57.421241    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:31:57.421307 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:57.421269    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:31:57.654227 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:57.654186    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg" event={"ID":"448a9f63-dbbb-4023-ba86-2760e97e5671","Type":"ContainerStarted","Data":"a160c0914de81c160ad0d0f541daafbdcd91df52b971d6b98160dcbdf890b4c6"}
Jul 16 12:31:57.654672 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:57.654251    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg" event={"ID":"448a9f63-dbbb-4023-ba86-2760e97e5671","Type":"ContainerStarted","Data":"8b7f9c333c207cccce3e1552f5804fb1f7aed8f3410c8bf8d59fcf1459b07f64"}
Jul 16 12:31:58.264079 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:58.264019    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:58.268974 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:58.268950    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:58.305911 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:58.305864    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg" podStartSLOduration=2.305847608 podStartE2EDuration="2.305847608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:31:57.682693879 +0000 UTC m=+2458.983470999" watchObservedRunningTime="2026-07-16 12:31:58.305847608 +0000 UTC m=+2459.606624707"
Jul 16 12:31:58.659586 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:58.659555    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:31:58.660618 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:31:58.660601    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:32:00.659762 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.659699    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:32:00.670351 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.670319    2578 generic.go:356] "Generic (PLEG): container finished" podID="baaf5878-af82-4b4f-85f4-157d901e8b22" containerID="acf16d00349ffb254b6e236240bfd2e726c6dc2d844e78d714a7101be486ecb8" exitCode=0
Jul 16 12:32:00.670470 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.670355    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm" event={"ID":"baaf5878-af82-4b4f-85f4-157d901e8b22","Type":"ContainerDied","Data":"acf16d00349ffb254b6e236240bfd2e726c6dc2d844e78d714a7101be486ecb8"}
Jul 16 12:32:00.670470 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.670369    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"
Jul 16 12:32:00.670470 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.670406    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm" event={"ID":"baaf5878-af82-4b4f-85f4-157d901e8b22","Type":"ContainerDied","Data":"4bd275d23b55f9d08572eac928d5ab7b090d7914849bbd68c8f94e3a37c1564b"}
Jul 16 12:32:00.670470 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.670432    2578 scope.go:117] "RemoveContainer" containerID="acf16d00349ffb254b6e236240bfd2e726c6dc2d844e78d714a7101be486ecb8"
Jul 16 12:32:00.679426 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.679404    2578 scope.go:117] "RemoveContainer" containerID="11fedbc4058804413344b9d7f5568ef5e95fb081d40d8158041f723fdf99f75b"
Jul 16 12:32:00.687047 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.687022    2578 scope.go:117] "RemoveContainer" containerID="acf16d00349ffb254b6e236240bfd2e726c6dc2d844e78d714a7101be486ecb8"
Jul 16 12:32:00.687304 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:32:00.687289    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acf16d00349ffb254b6e236240bfd2e726c6dc2d844e78d714a7101be486ecb8\": container with ID starting with acf16d00349ffb254b6e236240bfd2e726c6dc2d844e78d714a7101be486ecb8 not found: ID does not exist" containerID="acf16d00349ffb254b6e236240bfd2e726c6dc2d844e78d714a7101be486ecb8"
Jul 16 12:32:00.687366 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.687312    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acf16d00349ffb254b6e236240bfd2e726c6dc2d844e78d714a7101be486ecb8"} err="failed to get container status \"acf16d00349ffb254b6e236240bfd2e726c6dc2d844e78d714a7101be486ecb8\": rpc error: code = NotFound desc = could not find container \"acf16d00349ffb254b6e236240bfd2e726c6dc2d844e78d714a7101be486ecb8\": container with ID starting with acf16d00349ffb254b6e236240bfd2e726c6dc2d844e78d714a7101be486ecb8 not found: ID does not exist"
Jul 16 12:32:00.687366 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.687330    2578 scope.go:117] "RemoveContainer" containerID="11fedbc4058804413344b9d7f5568ef5e95fb081d40d8158041f723fdf99f75b"
Jul 16 12:32:00.687560 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:32:00.687542    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11fedbc4058804413344b9d7f5568ef5e95fb081d40d8158041f723fdf99f75b\": container with ID starting with 11fedbc4058804413344b9d7f5568ef5e95fb081d40d8158041f723fdf99f75b not found: ID does not exist" containerID="11fedbc4058804413344b9d7f5568ef5e95fb081d40d8158041f723fdf99f75b"
Jul 16 12:32:00.687605 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.687566    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11fedbc4058804413344b9d7f5568ef5e95fb081d40d8158041f723fdf99f75b"} err="failed to get container status \"11fedbc4058804413344b9d7f5568ef5e95fb081d40d8158041f723fdf99f75b\": rpc error: code = NotFound desc = could not find container \"11fedbc4058804413344b9d7f5568ef5e95fb081d40d8158041f723fdf99f75b\": container with ID starting with 11fedbc4058804413344b9d7f5568ef5e95fb081d40d8158041f723fdf99f75b not found: ID does not exist"
Jul 16 12:32:00.857207 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.857177    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-credential-socket\") pod \"baaf5878-af82-4b4f-85f4-157d901e8b22\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") "
Jul 16 12:32:00.857207 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.857219    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-token\") pod \"baaf5878-af82-4b4f-85f4-157d901e8b22\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") "
Jul 16 12:32:00.857467 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.857241    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-workload-socket\") pod \"baaf5878-af82-4b4f-85f4-157d901e8b22\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") "
Jul 16 12:32:00.857467 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.857305    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-data\") pod \"baaf5878-af82-4b4f-85f4-157d901e8b22\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") "
Jul 16 12:32:00.857467 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.857329    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr42d\" (UniqueName: \"kubernetes.io/projected/baaf5878-af82-4b4f-85f4-157d901e8b22-kube-api-access-wr42d\") pod \"baaf5878-af82-4b4f-85f4-157d901e8b22\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") "
Jul 16 12:32:00.857467 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.857374    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-envoy\") pod \"baaf5878-af82-4b4f-85f4-157d901e8b22\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") "
Jul 16 12:32:00.857467 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.857419    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-workload-certs\") pod \"baaf5878-af82-4b4f-85f4-157d901e8b22\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") "
Jul 16 12:32:00.857467 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.857446    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-podinfo\") pod \"baaf5878-af82-4b4f-85f4-157d901e8b22\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") "
Jul 16 12:32:00.857887 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.857489    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "baaf5878-af82-4b4f-85f4-157d901e8b22" (UID: "baaf5878-af82-4b4f-85f4-157d901e8b22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:32:00.857887 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.857505    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baaf5878-af82-4b4f-85f4-157d901e8b22-istiod-ca-cert\") pod \"baaf5878-af82-4b4f-85f4-157d901e8b22\" (UID: \"baaf5878-af82-4b4f-85f4-157d901e8b22\") "
Jul 16 12:32:00.857887 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.857565    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "baaf5878-af82-4b4f-85f4-157d901e8b22" (UID: "baaf5878-af82-4b4f-85f4-157d901e8b22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:32:00.857887 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.857850    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-credential-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:32:00.857887 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.857870    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-workload-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:32:00.858330 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.858093    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "baaf5878-af82-4b4f-85f4-157d901e8b22" (UID: "baaf5878-af82-4b4f-85f4-157d901e8b22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:32:00.858330 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.858113    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-data" (OuterVolumeSpecName: "istio-data") pod "baaf5878-af82-4b4f-85f4-157d901e8b22" (UID: "baaf5878-af82-4b4f-85f4-157d901e8b22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:32:00.858330 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.858254    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baaf5878-af82-4b4f-85f4-157d901e8b22-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "baaf5878-af82-4b4f-85f4-157d901e8b22" (UID: "baaf5878-af82-4b4f-85f4-157d901e8b22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 12:32:00.860228 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.860001    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-token" (OuterVolumeSpecName: "istio-token") pod "baaf5878-af82-4b4f-85f4-157d901e8b22" (UID: "baaf5878-af82-4b4f-85f4-157d901e8b22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:32:00.860228 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.860126    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "baaf5878-af82-4b4f-85f4-157d901e8b22" (UID: "baaf5878-af82-4b4f-85f4-157d901e8b22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:32:00.860409 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.860271    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baaf5878-af82-4b4f-85f4-157d901e8b22-kube-api-access-wr42d" (OuterVolumeSpecName: "kube-api-access-wr42d") pod "baaf5878-af82-4b4f-85f4-157d901e8b22" (UID: "baaf5878-af82-4b4f-85f4-157d901e8b22"). InnerVolumeSpecName "kube-api-access-wr42d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:32:00.860409 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.860395    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "baaf5878-af82-4b4f-85f4-157d901e8b22" (UID: "baaf5878-af82-4b4f-85f4-157d901e8b22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 12:32:00.958599 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.958513    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-data\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:32:00.958599 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.958542    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr42d\" (UniqueName: \"kubernetes.io/projected/baaf5878-af82-4b4f-85f4-157d901e8b22-kube-api-access-wr42d\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:32:00.958599 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.958552    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-envoy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:32:00.958599 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.958561    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baaf5878-af82-4b4f-85f4-157d901e8b22-workload-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:32:00.958599 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.958569    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-podinfo\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:32:00.958599 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.958576    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baaf5878-af82-4b4f-85f4-157d901e8b22-istiod-ca-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:32:00.958599 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.958584    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baaf5878-af82-4b4f-85f4-157d901e8b22-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:32:00.998218 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:00.998186    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"]
Jul 16 12:32:01.011121 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:01.011085    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e41619-openshift-default-96fff7b7-sbbrm"]
Jul 16 12:32:01.300618 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:01.300585    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baaf5878-af82-4b4f-85f4-157d901e8b22" path="/var/lib/kubelet/pods/baaf5878-af82-4b4f-85f4-157d901e8b22/volumes"
Jul 16 12:32:05.186950 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.186919    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"]
Jul 16 12:32:05.187385 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.187342    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baaf5878-af82-4b4f-85f4-157d901e8b22" containerName="istio-proxy"
Jul 16 12:32:05.187385 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.187355    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="baaf5878-af82-4b4f-85f4-157d901e8b22" containerName="istio-proxy"
Jul 16 12:32:05.187385 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.187379    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baaf5878-af82-4b4f-85f4-157d901e8b22" containerName="istio-proxy"
Jul 16 12:32:05.187385 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.187387    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="baaf5878-af82-4b4f-85f4-157d901e8b22" containerName="istio-proxy"
Jul 16 12:32:05.187518 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.187448    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="baaf5878-af82-4b4f-85f4-157d901e8b22" containerName="istio-proxy"
Jul 16 12:32:05.191718 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.191697    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.194818 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.194800    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-347f37-openshift-default-dockercfg-b6665\""
Jul 16 12:32:05.197632 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.197613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-podinfo\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.197728 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.197651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-data\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.197728 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.197675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-envoy\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.197841 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.197790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-workload-certs\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.197903 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.197885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-credential-socket\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.197948 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.197927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-workload-socket\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.197988 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.197949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a81f61-5072-4359-8b54-acb6f2c5f116-istiod-ca-cert\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.197988 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.197971    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g9ft\" (UniqueName: \"kubernetes.io/projected/a7a81f61-5072-4359-8b54-acb6f2c5f116-kube-api-access-4g9ft\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.198121 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.198031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-token\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.207154 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.207132    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"]
Jul 16 12:32:05.298648 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.298618    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-credential-socket\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.298824 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.298657    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-workload-socket\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.298824 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.298675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a81f61-5072-4359-8b54-acb6f2c5f116-istiod-ca-cert\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.298824 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.298691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g9ft\" (UniqueName: \"kubernetes.io/projected/a7a81f61-5072-4359-8b54-acb6f2c5f116-kube-api-access-4g9ft\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.298824 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.298718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-token\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.298824 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.298771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-podinfo\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.299103 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.299014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-data\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.299169 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.299106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-envoy\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.299169 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.299142    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-workload-socket\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.299276 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.299187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-workload-certs\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.299390 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.299368    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-data\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.299470 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.299439    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-credential-socket\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.299541 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.299520    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-workload-certs\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.299594 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.299522    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a81f61-5072-4359-8b54-acb6f2c5f116-istiod-ca-cert\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.301291 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.301266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-envoy\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.301463 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.301445    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-podinfo\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.308648 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.308628    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-token\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.309404 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.309380    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g9ft\" (UniqueName: \"kubernetes.io/projected/a7a81f61-5072-4359-8b54-acb6f2c5f116-kube-api-access-4g9ft\") pod \"e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") " pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.502399 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.502306    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:05.712677 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.712648    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"]
Jul 16 12:32:05.716404 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:32:05.716378    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a81f61_5072_4359_8b54_acb6f2c5f116.slice/crio-e10fa6dd1e9c3c4dca27d1ce295d822461487008ad0bae60386ff6892ba2aec9 WatchSource:0}: Error finding container e10fa6dd1e9c3c4dca27d1ce295d822461487008ad0bae60386ff6892ba2aec9: Status 404 returned error can't find the container with id e10fa6dd1e9c3c4dca27d1ce295d822461487008ad0bae60386ff6892ba2aec9
Jul 16 12:32:05.718629 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.718589    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:32:05.718691 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.718679    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:32:05.718748 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:05.718730    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:32:06.704546 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:06.704381    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k" event={"ID":"a7a81f61-5072-4359-8b54-acb6f2c5f116","Type":"ContainerStarted","Data":"855abd5fa0a5e6a2c5ef93e0b366de5441f0726e40360c14041f1aff6bddb0cf"}
Jul 16 12:32:06.705005 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:06.704902    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k" event={"ID":"a7a81f61-5072-4359-8b54-acb6f2c5f116","Type":"ContainerStarted","Data":"e10fa6dd1e9c3c4dca27d1ce295d822461487008ad0bae60386ff6892ba2aec9"}
Jul 16 12:32:07.502832 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:07.502797    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:07.507493 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:07.507469    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:07.539057 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:07.538333    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k" podStartSLOduration=2.538313049 podStartE2EDuration="2.538313049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:32:06.758335812 +0000 UTC m=+2468.059112923" watchObservedRunningTime="2026-07-16 12:32:07.538313049 +0000 UTC m=+2468.839090148"
Jul 16 12:32:07.709654 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:07.709620    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:07.710598 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:07.710580    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:12.735864 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:12.735828    2578 generic.go:356] "Generic (PLEG): container finished" podID="448a9f63-dbbb-4023-ba86-2760e97e5671" containerID="a160c0914de81c160ad0d0f541daafbdcd91df52b971d6b98160dcbdf890b4c6" exitCode=0
Jul 16 12:32:12.736350 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:12.735907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg" event={"ID":"448a9f63-dbbb-4023-ba86-2760e97e5671","Type":"ContainerDied","Data":"a160c0914de81c160ad0d0f541daafbdcd91df52b971d6b98160dcbdf890b4c6"}
Jul 16 12:32:12.736350 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:12.736318    2578 scope.go:117] "RemoveContainer" containerID="a160c0914de81c160ad0d0f541daafbdcd91df52b971d6b98160dcbdf890b4c6"
Jul 16 12:32:12.737330 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:12.737301    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:32:12.737394 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:12.737368    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:32:12.737438 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:12.737399    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:32:13.659870 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:13.659828    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:32:13.746544 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:13.746506    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg" event={"ID":"448a9f63-dbbb-4023-ba86-2760e97e5671","Type":"ContainerStarted","Data":"555d9fc92ec8aba1b62148e24284f795513a918e40bcf7b59871ff9ac7207e54"}
Jul 16 12:32:14.264134 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:14.264094    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:32:14.265366 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:14.265338    2578 patch_prober.go:28] interesting pod/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 16 12:32:14.265494 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:14.265398    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg" podUID="448a9f63-dbbb-4023-ba86-2760e97e5671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 16 12:32:15.265352 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:15.265314    2578 patch_prober.go:28] interesting pod/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 16 12:32:15.265934 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:15.265903    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg" podUID="448a9f63-dbbb-4023-ba86-2760e97e5671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 16 12:32:16.263947 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:16.263908    2578 patch_prober.go:28] interesting pod/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 16 12:32:16.264259 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:16.264231    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg" podUID="448a9f63-dbbb-4023-ba86-2760e97e5671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 16 12:32:17.264571 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:17.264527    2578 patch_prober.go:28] interesting pod/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 16 12:32:17.265213 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:17.265183    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg" podUID="448a9f63-dbbb-4023-ba86-2760e97e5671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 16 12:32:18.267900 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:18.267864    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:32:18.268395 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:18.268250    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:32:18.268800 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:18.268780    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:32:22.711481 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:22.711446    2578 patch_prober.go:28] interesting pod/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Jul 16 12:32:22.711896 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:22.711505    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k" podUID="a7a81f61-5072-4359-8b54-acb6f2c5f116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Jul 16 12:32:22.798371 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:22.798339    2578 generic.go:356] "Generic (PLEG): container finished" podID="a7a81f61-5072-4359-8b54-acb6f2c5f116" containerID="855abd5fa0a5e6a2c5ef93e0b366de5441f0726e40360c14041f1aff6bddb0cf" exitCode=0
Jul 16 12:32:22.798546 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:22.798422    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k" event={"ID":"a7a81f61-5072-4359-8b54-acb6f2c5f116","Type":"ContainerDied","Data":"855abd5fa0a5e6a2c5ef93e0b366de5441f0726e40360c14041f1aff6bddb0cf"}
Jul 16 12:32:22.798866 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:22.798852    2578 scope.go:117] "RemoveContainer" containerID="855abd5fa0a5e6a2c5ef93e0b366de5441f0726e40360c14041f1aff6bddb0cf"
Jul 16 12:32:22.799857 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:22.799829    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:32:22.799925 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:22.799898    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:32:22.799967 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:22.799931    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:32:23.805242 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:23.805210    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k" event={"ID":"a7a81f61-5072-4359-8b54-acb6f2c5f116","Type":"ContainerStarted","Data":"480e07f81a288bab8d0e989e3e3db87f7f41e048e7a67d4f0e597bffb32b193e"}
Jul 16 12:32:24.503119 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:24.503085    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:24.503341 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:24.503318    2578 patch_prober.go:28] interesting pod/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Jul 16 12:32:24.503391 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:24.503369    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k" podUID="a7a81f61-5072-4359-8b54-acb6f2c5f116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Jul 16 12:32:25.502944 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:25.502904    2578 patch_prober.go:28] interesting pod/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Jul 16 12:32:25.503400 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:25.502964    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k" podUID="a7a81f61-5072-4359-8b54-acb6f2c5f116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Jul 16 12:32:26.502957 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:26.502924    2578 patch_prober.go:28] interesting pod/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Jul 16 12:32:26.503373 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:26.502976    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k" podUID="a7a81f61-5072-4359-8b54-acb6f2c5f116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Jul 16 12:32:27.502935 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:27.502905    2578 patch_prober.go:28] interesting pod/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Jul 16 12:32:27.503118 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:27.502966    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k" podUID="a7a81f61-5072-4359-8b54-acb6f2c5f116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Jul 16 12:32:28.506565 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:28.506537    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:28.506921 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:28.506889    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:28.507475 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:28.507456    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:32:47.922541 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:47.922506    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"]
Jul 16 12:32:47.923008 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:47.922697    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg" podUID="448a9f63-dbbb-4023-ba86-2760e97e5671" containerName="istio-proxy" containerID="cri-o://555d9fc92ec8aba1b62148e24284f795513a918e40bcf7b59871ff9ac7207e54" gracePeriod=30
Jul 16 12:32:48.271032 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:48.270948    2578 patch_prober.go:28] interesting pod/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 12:32:48.271032 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:48.271008    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg" podUID="448a9f63-dbbb-4023-ba86-2760e97e5671" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 12:32:53.179316 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.179284    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:32:53.254891 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.254792    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-credential-socket\") pod \"448a9f63-dbbb-4023-ba86-2760e97e5671\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") "
Jul 16 12:32:53.254891 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.254831    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-envoy\") pod \"448a9f63-dbbb-4023-ba86-2760e97e5671\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") "
Jul 16 12:32:53.254891 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.254889    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvfnb\" (UniqueName: \"kubernetes.io/projected/448a9f63-dbbb-4023-ba86-2760e97e5671-kube-api-access-nvfnb\") pod \"448a9f63-dbbb-4023-ba86-2760e97e5671\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") "
Jul 16 12:32:53.255218 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.254915    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-podinfo\") pod \"448a9f63-dbbb-4023-ba86-2760e97e5671\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") "
Jul 16 12:32:53.255218 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.254947    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/448a9f63-dbbb-4023-ba86-2760e97e5671-istiod-ca-cert\") pod \"448a9f63-dbbb-4023-ba86-2760e97e5671\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") "
Jul 16 12:32:53.255218 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.254976    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-workload-socket\") pod \"448a9f63-dbbb-4023-ba86-2760e97e5671\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") "
Jul 16 12:32:53.255218 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.254999    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-token\") pod \"448a9f63-dbbb-4023-ba86-2760e97e5671\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") "
Jul 16 12:32:53.255218 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.255063    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-data\") pod \"448a9f63-dbbb-4023-ba86-2760e97e5671\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") "
Jul 16 12:32:53.255218 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.255088    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-workload-certs\") pod \"448a9f63-dbbb-4023-ba86-2760e97e5671\" (UID: \"448a9f63-dbbb-4023-ba86-2760e97e5671\") "
Jul 16 12:32:53.255218 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.255149    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "448a9f63-dbbb-4023-ba86-2760e97e5671" (UID: "448a9f63-dbbb-4023-ba86-2760e97e5671"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:32:53.255565 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.255281    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "448a9f63-dbbb-4023-ba86-2760e97e5671" (UID: "448a9f63-dbbb-4023-ba86-2760e97e5671"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:32:53.255565 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.255413    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/448a9f63-dbbb-4023-ba86-2760e97e5671-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "448a9f63-dbbb-4023-ba86-2760e97e5671" (UID: "448a9f63-dbbb-4023-ba86-2760e97e5671"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 12:32:53.255565 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.255518    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "448a9f63-dbbb-4023-ba86-2760e97e5671" (UID: "448a9f63-dbbb-4023-ba86-2760e97e5671"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:32:53.255565 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.255539    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-credential-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:32:53.255565 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.255559    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-workload-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:32:53.256358 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.256330    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-data" (OuterVolumeSpecName: "istio-data") pod "448a9f63-dbbb-4023-ba86-2760e97e5671" (UID: "448a9f63-dbbb-4023-ba86-2760e97e5671"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:32:53.257294 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.257265    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-token" (OuterVolumeSpecName: "istio-token") pod "448a9f63-dbbb-4023-ba86-2760e97e5671" (UID: "448a9f63-dbbb-4023-ba86-2760e97e5671"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:32:53.257294 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.257279    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "448a9f63-dbbb-4023-ba86-2760e97e5671" (UID: "448a9f63-dbbb-4023-ba86-2760e97e5671"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:32:53.257417 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.257314    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448a9f63-dbbb-4023-ba86-2760e97e5671-kube-api-access-nvfnb" (OuterVolumeSpecName: "kube-api-access-nvfnb") pod "448a9f63-dbbb-4023-ba86-2760e97e5671" (UID: "448a9f63-dbbb-4023-ba86-2760e97e5671"). InnerVolumeSpecName "kube-api-access-nvfnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:32:53.257417 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.257384    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "448a9f63-dbbb-4023-ba86-2760e97e5671" (UID: "448a9f63-dbbb-4023-ba86-2760e97e5671"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 12:32:53.356447 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.356408    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-data\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:32:53.356447 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.356445    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-workload-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:32:53.356649 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.356459    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-envoy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:32:53.356649 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.356472    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvfnb\" (UniqueName: \"kubernetes.io/projected/448a9f63-dbbb-4023-ba86-2760e97e5671-kube-api-access-nvfnb\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:32:53.356649 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.356485    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-podinfo\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:32:53.356649 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.356499    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/448a9f63-dbbb-4023-ba86-2760e97e5671-istiod-ca-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:32:53.356649 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.356512    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/448a9f63-dbbb-4023-ba86-2760e97e5671-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:32:53.957819 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.957785    2578 generic.go:356] "Generic (PLEG): container finished" podID="448a9f63-dbbb-4023-ba86-2760e97e5671" containerID="555d9fc92ec8aba1b62148e24284f795513a918e40bcf7b59871ff9ac7207e54" exitCode=0
Jul 16 12:32:53.957998 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.957829    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"
Jul 16 12:32:53.957998 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.957867    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg" event={"ID":"448a9f63-dbbb-4023-ba86-2760e97e5671","Type":"ContainerDied","Data":"555d9fc92ec8aba1b62148e24284f795513a918e40bcf7b59871ff9ac7207e54"}
Jul 16 12:32:53.957998 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.957923    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg" event={"ID":"448a9f63-dbbb-4023-ba86-2760e97e5671","Type":"ContainerDied","Data":"8b7f9c333c207cccce3e1552f5804fb1f7aed8f3410c8bf8d59fcf1459b07f64"}
Jul 16 12:32:53.957998 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.957945    2578 scope.go:117] "RemoveContainer" containerID="555d9fc92ec8aba1b62148e24284f795513a918e40bcf7b59871ff9ac7207e54"
Jul 16 12:32:53.966756 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.966738    2578 scope.go:117] "RemoveContainer" containerID="a160c0914de81c160ad0d0f541daafbdcd91df52b971d6b98160dcbdf890b4c6"
Jul 16 12:32:53.974678 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.974660    2578 scope.go:117] "RemoveContainer" containerID="555d9fc92ec8aba1b62148e24284f795513a918e40bcf7b59871ff9ac7207e54"
Jul 16 12:32:53.974967 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:32:53.974945    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"555d9fc92ec8aba1b62148e24284f795513a918e40bcf7b59871ff9ac7207e54\": container with ID starting with 555d9fc92ec8aba1b62148e24284f795513a918e40bcf7b59871ff9ac7207e54 not found: ID does not exist" containerID="555d9fc92ec8aba1b62148e24284f795513a918e40bcf7b59871ff9ac7207e54"
Jul 16 12:32:53.975076 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.974978    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"555d9fc92ec8aba1b62148e24284f795513a918e40bcf7b59871ff9ac7207e54"} err="failed to get container status \"555d9fc92ec8aba1b62148e24284f795513a918e40bcf7b59871ff9ac7207e54\": rpc error: code = NotFound desc = could not find container \"555d9fc92ec8aba1b62148e24284f795513a918e40bcf7b59871ff9ac7207e54\": container with ID starting with 555d9fc92ec8aba1b62148e24284f795513a918e40bcf7b59871ff9ac7207e54 not found: ID does not exist"
Jul 16 12:32:53.975076 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.975003    2578 scope.go:117] "RemoveContainer" containerID="a160c0914de81c160ad0d0f541daafbdcd91df52b971d6b98160dcbdf890b4c6"
Jul 16 12:32:53.975310 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:32:53.975292    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a160c0914de81c160ad0d0f541daafbdcd91df52b971d6b98160dcbdf890b4c6\": container with ID starting with a160c0914de81c160ad0d0f541daafbdcd91df52b971d6b98160dcbdf890b4c6 not found: ID does not exist" containerID="a160c0914de81c160ad0d0f541daafbdcd91df52b971d6b98160dcbdf890b4c6"
Jul 16 12:32:53.975365 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.975319    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a160c0914de81c160ad0d0f541daafbdcd91df52b971d6b98160dcbdf890b4c6"} err="failed to get container status \"a160c0914de81c160ad0d0f541daafbdcd91df52b971d6b98160dcbdf890b4c6\": rpc error: code = NotFound desc = could not find container \"a160c0914de81c160ad0d0f541daafbdcd91df52b971d6b98160dcbdf890b4c6\": container with ID starting with a160c0914de81c160ad0d0f541daafbdcd91df52b971d6b98160dcbdf890b4c6 not found: ID does not exist"
Jul 16 12:32:53.976498 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.976481    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"]
Jul 16 12:32:53.981386 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:53.981362    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2f272c-openshift-default-5fbd9b9ff5-8ktwg"]
Jul 16 12:32:55.299724 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:32:55.299695    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="448a9f63-dbbb-4023-ba86-2760e97e5671" path="/var/lib/kubelet/pods/448a9f63-dbbb-4023-ba86-2760e97e5671/volumes"
Jul 16 12:33:14.256413 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:14.256380    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"]
Jul 16 12:33:14.256970 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:14.256576    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k" podUID="a7a81f61-5072-4359-8b54-acb6f2c5f116" containerName="istio-proxy" containerID="cri-o://480e07f81a288bab8d0e989e3e3db87f7f41e048e7a67d4f0e597bffb32b193e" gracePeriod=30
Jul 16 12:33:15.237514 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.237471    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"]
Jul 16 12:33:15.238028 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.238012    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="448a9f63-dbbb-4023-ba86-2760e97e5671" containerName="istio-proxy"
Jul 16 12:33:15.238106 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.238032    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="448a9f63-dbbb-4023-ba86-2760e97e5671" containerName="istio-proxy"
Jul 16 12:33:15.238106 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.238085    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="448a9f63-dbbb-4023-ba86-2760e97e5671" containerName="istio-proxy"
Jul 16 12:33:15.238106 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.238093    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="448a9f63-dbbb-4023-ba86-2760e97e5671" containerName="istio-proxy"
Jul 16 12:33:15.238205 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.238164    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="448a9f63-dbbb-4023-ba86-2760e97e5671" containerName="istio-proxy"
Jul 16 12:33:15.238205 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.238173    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="baaf5878-af82-4b4f-85f4-157d901e8b22" containerName="istio-proxy"
Jul 16 12:33:15.241129 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.241113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.244295 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.244278    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-a96583-openshift-default-dockercfg-5fmwh\""
Jul 16 12:33:15.255593 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.255570    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"]
Jul 16 12:33:15.358386 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.358347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz7vj\" (UniqueName: \"kubernetes.io/projected/154fa342-6acf-4a63-832c-b78443bc7b17-kube-api-access-dz7vj\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.358386 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.358392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-workload-certs\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.358821 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.358412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-istio-envoy\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.358821 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.358496    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154fa342-6acf-4a63-832c-b78443bc7b17-istio-token\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.358821 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.358538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-credential-socket\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.358821 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.358563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-istio-data\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.358821 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.358623    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154fa342-6acf-4a63-832c-b78443bc7b17-istio-podinfo\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.358821 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.358646    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-workload-socket\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.358821 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.358660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154fa342-6acf-4a63-832c-b78443bc7b17-istiod-ca-cert\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.459385 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.459344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz7vj\" (UniqueName: \"kubernetes.io/projected/154fa342-6acf-4a63-832c-b78443bc7b17-kube-api-access-dz7vj\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.459385 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.459387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-workload-certs\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.459658 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.459412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-istio-envoy\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.459658 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.459451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154fa342-6acf-4a63-832c-b78443bc7b17-istio-token\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.459658 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.459504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-credential-socket\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.459658 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.459596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-istio-data\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.459658 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.459656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154fa342-6acf-4a63-832c-b78443bc7b17-istio-podinfo\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.459907 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.459706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-workload-socket\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.459907 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.459731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154fa342-6acf-4a63-832c-b78443bc7b17-istiod-ca-cert\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.459907 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.459807    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-workload-certs\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.459907 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.459836    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-credential-socket\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.460246 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.460217    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-istio-data\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.460384 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.460274    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-workload-socket\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.460527 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.460503    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154fa342-6acf-4a63-832c-b78443bc7b17-istiod-ca-cert\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.462295 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.462271    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-istio-envoy\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.462494 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.462478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154fa342-6acf-4a63-832c-b78443bc7b17-istio-podinfo\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.475935 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.475903    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154fa342-6acf-4a63-832c-b78443bc7b17-istio-token\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.476224 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.476198    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz7vj\" (UniqueName: \"kubernetes.io/projected/154fa342-6acf-4a63-832c-b78443bc7b17-kube-api-access-dz7vj\") pod \"e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") " pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.551205 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.551176    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:15.705790 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.705762    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"]
Jul 16 12:33:15.706897 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:33:15.706869    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154fa342_6acf_4a63_832c_b78443bc7b17.slice/crio-c2a36607575a9a9efba170928deed6748a3f939fbe8f94bf5d4f42ba93ab2e05 WatchSource:0}: Error finding container c2a36607575a9a9efba170928deed6748a3f939fbe8f94bf5d4f42ba93ab2e05: Status 404 returned error can't find the container with id c2a36607575a9a9efba170928deed6748a3f939fbe8f94bf5d4f42ba93ab2e05
Jul 16 12:33:15.709393 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.709359    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:33:15.709511 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.709436    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:33:15.709511 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:15.709482    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:33:16.063898 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:16.063867    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6" event={"ID":"154fa342-6acf-4a63-832c-b78443bc7b17","Type":"ContainerStarted","Data":"b5215a1939469271acc5d344093b3152c722a7060cd4951ba92e1747498ea9ab"}
Jul 16 12:33:16.064092 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:16.063917    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6" event={"ID":"154fa342-6acf-4a63-832c-b78443bc7b17","Type":"ContainerStarted","Data":"c2a36607575a9a9efba170928deed6748a3f939fbe8f94bf5d4f42ba93ab2e05"}
Jul 16 12:33:16.096968 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:16.096918    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6" podStartSLOduration=1.096900084 podStartE2EDuration="1.096900084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:33:16.095632915 +0000 UTC m=+2537.396410029" watchObservedRunningTime="2026-07-16 12:33:16.096900084 +0000 UTC m=+2537.397677183"
Jul 16 12:33:16.552142 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:16.552110    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:16.557144 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:16.557124    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:17.068526 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:17.068494    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:17.069540 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:17.069522    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:19.519639 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.519575    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:33:19.698508 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.698466    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-credential-socket\") pod \"a7a81f61-5072-4359-8b54-acb6f2c5f116\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") "
Jul 16 12:33:19.698673 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.698555    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-data\") pod \"a7a81f61-5072-4359-8b54-acb6f2c5f116\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") "
Jul 16 12:33:19.698673 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.698584    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-workload-socket\") pod \"a7a81f61-5072-4359-8b54-acb6f2c5f116\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") "
Jul 16 12:33:19.698673 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.698609    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a81f61-5072-4359-8b54-acb6f2c5f116-istiod-ca-cert\") pod \"a7a81f61-5072-4359-8b54-acb6f2c5f116\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") "
Jul 16 12:33:19.698673 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.698669    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-podinfo\") pod \"a7a81f61-5072-4359-8b54-acb6f2c5f116\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") "
Jul 16 12:33:19.698882 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.698694    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-token\") pod \"a7a81f61-5072-4359-8b54-acb6f2c5f116\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") "
Jul 16 12:33:19.698882 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.698710    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7a81f61-5072-4359-8b54-acb6f2c5f116" (UID: "a7a81f61-5072-4359-8b54-acb6f2c5f116"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:33:19.698882 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.698838    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-envoy\") pod \"a7a81f61-5072-4359-8b54-acb6f2c5f116\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") "
Jul 16 12:33:19.698882 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.698865    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7a81f61-5072-4359-8b54-acb6f2c5f116" (UID: "a7a81f61-5072-4359-8b54-acb6f2c5f116"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:33:19.698882 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.698879    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-workload-certs\") pod \"a7a81f61-5072-4359-8b54-acb6f2c5f116\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") "
Jul 16 12:33:19.699202 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.698965    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g9ft\" (UniqueName: \"kubernetes.io/projected/a7a81f61-5072-4359-8b54-acb6f2c5f116-kube-api-access-4g9ft\") pod \"a7a81f61-5072-4359-8b54-acb6f2c5f116\" (UID: \"a7a81f61-5072-4359-8b54-acb6f2c5f116\") "
Jul 16 12:33:19.699202 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.699176    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a81f61-5072-4359-8b54-acb6f2c5f116-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7a81f61-5072-4359-8b54-acb6f2c5f116" (UID: "a7a81f61-5072-4359-8b54-acb6f2c5f116"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 12:33:19.699392 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.699368    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-workload-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:33:19.699460 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.699405    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a81f61-5072-4359-8b54-acb6f2c5f116-istiod-ca-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:33:19.699460 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.699421    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-credential-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:33:19.699460 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.699365    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7a81f61-5072-4359-8b54-acb6f2c5f116" (UID: "a7a81f61-5072-4359-8b54-acb6f2c5f116"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:33:19.699623 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.699571    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7a81f61-5072-4359-8b54-acb6f2c5f116" (UID: "a7a81f61-5072-4359-8b54-acb6f2c5f116"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:33:19.701219 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.701180    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7a81f61-5072-4359-8b54-acb6f2c5f116" (UID: "a7a81f61-5072-4359-8b54-acb6f2c5f116"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:33:19.701338 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.701243    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a81f61-5072-4359-8b54-acb6f2c5f116-kube-api-access-4g9ft" (OuterVolumeSpecName: "kube-api-access-4g9ft") pod "a7a81f61-5072-4359-8b54-acb6f2c5f116" (UID: "a7a81f61-5072-4359-8b54-acb6f2c5f116"). InnerVolumeSpecName "kube-api-access-4g9ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:33:19.701455 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.701436    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7a81f61-5072-4359-8b54-acb6f2c5f116" (UID: "a7a81f61-5072-4359-8b54-acb6f2c5f116"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 12:33:19.701651 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.701636    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7a81f61-5072-4359-8b54-acb6f2c5f116" (UID: "a7a81f61-5072-4359-8b54-acb6f2c5f116"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:33:19.800736 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.800707    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-podinfo\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:33:19.800736 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.800733    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:33:19.800736 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.800742    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-envoy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:33:19.800986 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.800750    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-workload-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:33:19.800986 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.800758    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g9ft\" (UniqueName: \"kubernetes.io/projected/a7a81f61-5072-4359-8b54-acb6f2c5f116-kube-api-access-4g9ft\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:33:19.800986 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:19.800769    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a81f61-5072-4359-8b54-acb6f2c5f116-istio-data\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:33:20.083426 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:20.083338    2578 generic.go:356] "Generic (PLEG): container finished" podID="a7a81f61-5072-4359-8b54-acb6f2c5f116" containerID="480e07f81a288bab8d0e989e3e3db87f7f41e048e7a67d4f0e597bffb32b193e" exitCode=0
Jul 16 12:33:20.083426 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:20.083398    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"
Jul 16 12:33:20.083426 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:20.083401    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k" event={"ID":"a7a81f61-5072-4359-8b54-acb6f2c5f116","Type":"ContainerDied","Data":"480e07f81a288bab8d0e989e3e3db87f7f41e048e7a67d4f0e597bffb32b193e"}
Jul 16 12:33:20.083653 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:20.083457    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k" event={"ID":"a7a81f61-5072-4359-8b54-acb6f2c5f116","Type":"ContainerDied","Data":"e10fa6dd1e9c3c4dca27d1ce295d822461487008ad0bae60386ff6892ba2aec9"}
Jul 16 12:33:20.083653 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:20.083489    2578 scope.go:117] "RemoveContainer" containerID="480e07f81a288bab8d0e989e3e3db87f7f41e048e7a67d4f0e597bffb32b193e"
Jul 16 12:33:20.095088 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:20.095068    2578 scope.go:117] "RemoveContainer" containerID="855abd5fa0a5e6a2c5ef93e0b366de5441f0726e40360c14041f1aff6bddb0cf"
Jul 16 12:33:20.103719 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:20.103697    2578 scope.go:117] "RemoveContainer" containerID="480e07f81a288bab8d0e989e3e3db87f7f41e048e7a67d4f0e597bffb32b193e"
Jul 16 12:33:20.104079 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:33:20.104060    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"480e07f81a288bab8d0e989e3e3db87f7f41e048e7a67d4f0e597bffb32b193e\": container with ID starting with 480e07f81a288bab8d0e989e3e3db87f7f41e048e7a67d4f0e597bffb32b193e not found: ID does not exist" containerID="480e07f81a288bab8d0e989e3e3db87f7f41e048e7a67d4f0e597bffb32b193e"
Jul 16 12:33:20.104155 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:20.104087    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"480e07f81a288bab8d0e989e3e3db87f7f41e048e7a67d4f0e597bffb32b193e"} err="failed to get container status \"480e07f81a288bab8d0e989e3e3db87f7f41e048e7a67d4f0e597bffb32b193e\": rpc error: code = NotFound desc = could not find container \"480e07f81a288bab8d0e989e3e3db87f7f41e048e7a67d4f0e597bffb32b193e\": container with ID starting with 480e07f81a288bab8d0e989e3e3db87f7f41e048e7a67d4f0e597bffb32b193e not found: ID does not exist"
Jul 16 12:33:20.104155 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:20.104104    2578 scope.go:117] "RemoveContainer" containerID="855abd5fa0a5e6a2c5ef93e0b366de5441f0726e40360c14041f1aff6bddb0cf"
Jul 16 12:33:20.104376 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:33:20.104354    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"855abd5fa0a5e6a2c5ef93e0b366de5441f0726e40360c14041f1aff6bddb0cf\": container with ID starting with 855abd5fa0a5e6a2c5ef93e0b366de5441f0726e40360c14041f1aff6bddb0cf not found: ID does not exist" containerID="855abd5fa0a5e6a2c5ef93e0b366de5441f0726e40360c14041f1aff6bddb0cf"
Jul 16 12:33:20.104422 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:20.104386    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"855abd5fa0a5e6a2c5ef93e0b366de5441f0726e40360c14041f1aff6bddb0cf"} err="failed to get container status \"855abd5fa0a5e6a2c5ef93e0b366de5441f0726e40360c14041f1aff6bddb0cf\": rpc error: code = NotFound desc = could not find container \"855abd5fa0a5e6a2c5ef93e0b366de5441f0726e40360c14041f1aff6bddb0cf\": container with ID starting with 855abd5fa0a5e6a2c5ef93e0b366de5441f0726e40360c14041f1aff6bddb0cf not found: ID does not exist"
Jul 16 12:33:20.134019 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:20.133980    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"]
Jul 16 12:33:20.145601 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:20.145571    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-347f37-openshift-default-6655df8dbf-2d92k"]
Jul 16 12:33:21.299660 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:21.299633    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7a81f61-5072-4359-8b54-acb6f2c5f116" path="/var/lib/kubelet/pods/a7a81f61-5072-4359-8b54-acb6f2c5f116/volumes"
Jul 16 12:33:23.447451 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.447415    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"]
Jul 16 12:33:23.447812 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.447783    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a81f61-5072-4359-8b54-acb6f2c5f116" containerName="istio-proxy"
Jul 16 12:33:23.447812 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.447794    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a81f61-5072-4359-8b54-acb6f2c5f116" containerName="istio-proxy"
Jul 16 12:33:23.447891 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.447861    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7a81f61-5072-4359-8b54-acb6f2c5f116" containerName="istio-proxy"
Jul 16 12:33:23.447932 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.447871    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="448a9f63-dbbb-4023-ba86-2760e97e5671" containerName="istio-proxy"
Jul 16 12:33:23.447932 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.447902    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7a81f61-5072-4359-8b54-acb6f2c5f116" containerName="istio-proxy"
Jul 16 12:33:23.452487 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.452457    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.455269 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.455252    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-66e9a3-openshift-default-dockercfg-gm87f\""
Jul 16 12:33:23.489689 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.489665    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"]
Jul 16 12:33:23.636738 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.636705    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-credential-socket\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.636738 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.636736    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-data\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.636936 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.636778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-workload-socket\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.636936 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.636836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-envoy\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.636936 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.636870    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-token\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.636936 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.636898    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-podinfo\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.636936 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.636917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mclw2\" (UniqueName: \"kubernetes.io/projected/f45abf0a-af01-4622-8da8-ebd7d79c1bca-kube-api-access-mclw2\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.637123 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.636999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istiod-ca-cert\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.637123 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.637025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-workload-certs\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.738254 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.738169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istiod-ca-cert\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.738254 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.738205    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-workload-certs\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.738254 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.738239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-credential-socket\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.738254 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.738255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-data\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.738578 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.738292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-workload-socket\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.738578 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.738324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-envoy\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.738578 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.738341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-token\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.738578 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.738365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-podinfo\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.738578 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.738391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mclw2\" (UniqueName: \"kubernetes.io/projected/f45abf0a-af01-4622-8da8-ebd7d79c1bca-kube-api-access-mclw2\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.738831 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.738680    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-workload-certs\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.738928 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.738908    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istiod-ca-cert\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.739051 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.739014    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-credential-socket\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.739115 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.739063    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-data\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.739171 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.739122    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-workload-socket\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.741172 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.741149    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-envoy\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.741362 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.741345    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-podinfo\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.748887 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.748863    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-token\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.749214 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.749197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mclw2\" (UniqueName: \"kubernetes.io/projected/f45abf0a-af01-4622-8da8-ebd7d79c1bca-kube-api-access-mclw2\") pod \"e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") " pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.762346 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.762328    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:23.920679 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.920639    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"]
Jul 16 12:33:23.926160 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:33:23.926125    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45abf0a_af01_4622_8da8_ebd7d79c1bca.slice/crio-0d2c6d7554cc11365426d1c0e0537a38d55a5f8e7a328fe2ef0dc59369e9de7c WatchSource:0}: Error finding container 0d2c6d7554cc11365426d1c0e0537a38d55a5f8e7a328fe2ef0dc59369e9de7c: Status 404 returned error can't find the container with id 0d2c6d7554cc11365426d1c0e0537a38d55a5f8e7a328fe2ef0dc59369e9de7c
Jul 16 12:33:23.928885 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.928854    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:33:23.928986 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.928921    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:33:23.928986 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:23.928949    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:33:24.106436 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:24.106383    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl" event={"ID":"f45abf0a-af01-4622-8da8-ebd7d79c1bca","Type":"ContainerStarted","Data":"e2e7741462a31973a645893587a154f636a20bedf6374728a36bb98a8be2d259"}
Jul 16 12:33:24.106605 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:24.106500    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl" event={"ID":"f45abf0a-af01-4622-8da8-ebd7d79c1bca","Type":"ContainerStarted","Data":"0d2c6d7554cc11365426d1c0e0537a38d55a5f8e7a328fe2ef0dc59369e9de7c"}
Jul 16 12:33:24.137002 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:24.136924    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl" podStartSLOduration=1.136904779 podStartE2EDuration="1.136904779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:33:24.1351138 +0000 UTC m=+2545.435890899" watchObservedRunningTime="2026-07-16 12:33:24.136904779 +0000 UTC m=+2545.437681872"
Jul 16 12:33:24.762995 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:24.762962    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:24.764669 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:24.764641    2578 patch_prober.go:28] interesting pod/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 16 12:33:24.764792 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:24.764697    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl" podUID="f45abf0a-af01-4622-8da8-ebd7d79c1bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 16 12:33:25.766610 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:25.766580    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:26.115868 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:26.115838    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:26.116755 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:26.116737    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:31.142554 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:31.142465    2578 generic.go:356] "Generic (PLEG): container finished" podID="154fa342-6acf-4a63-832c-b78443bc7b17" containerID="b5215a1939469271acc5d344093b3152c722a7060cd4951ba92e1747498ea9ab" exitCode=0
Jul 16 12:33:31.142554 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:31.142537    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6" event={"ID":"154fa342-6acf-4a63-832c-b78443bc7b17","Type":"ContainerDied","Data":"b5215a1939469271acc5d344093b3152c722a7060cd4951ba92e1747498ea9ab"}
Jul 16 12:33:31.142998 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:31.142961    2578 scope.go:117] "RemoveContainer" containerID="b5215a1939469271acc5d344093b3152c722a7060cd4951ba92e1747498ea9ab"
Jul 16 12:33:31.144068 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:31.144022    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:33:31.144175 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:31.144109    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:33:31.144175 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:31.144141    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:33:32.069020 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:32.068971    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:32.152570 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:32.152533    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6" event={"ID":"154fa342-6acf-4a63-832c-b78443bc7b17","Type":"ContainerStarted","Data":"6a304db283ae222a7e5af5a80e3c6ddc638fa3f58111e0464a8570bc8aafac0a"}
Jul 16 12:33:32.552260 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:32.552222    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:32.553497 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:32.553464    2578 patch_prober.go:28] interesting pod/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 16 12:33:32.553654 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:32.553527    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6" podUID="154fa342-6acf-4a63-832c-b78443bc7b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 16 12:33:33.552158 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:33.552117    2578 patch_prober.go:28] interesting pod/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 16 12:33:33.552645 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:33.552189    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6" podUID="154fa342-6acf-4a63-832c-b78443bc7b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 16 12:33:34.552516 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:34.552482    2578 patch_prober.go:28] interesting pod/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 16 12:33:34.553129 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:34.553097    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6" podUID="154fa342-6acf-4a63-832c-b78443bc7b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 16 12:33:35.552180 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:35.552134    2578 patch_prober.go:28] interesting pod/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 16 12:33:35.552396 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:35.552204    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6" podUID="154fa342-6acf-4a63-832c-b78443bc7b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 16 12:33:36.555779 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:36.555746    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:36.556264 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:36.556138    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:36.556890 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:36.556867    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:33:40.882195 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:40.882160    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"]
Jul 16 12:33:40.882677 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:40.882563    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a81f61-5072-4359-8b54-acb6f2c5f116" containerName="istio-proxy"
Jul 16 12:33:40.882677 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:40.882575    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a81f61-5072-4359-8b54-acb6f2c5f116" containerName="istio-proxy"
Jul 16 12:33:40.886195 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:40.886174    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:40.888764 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:40.888737    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a96583\"/\"kube-root-ca.crt\""
Jul 16 12:33:40.889772 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:40.889493    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a96583\"/\"openshift-service-ca.crt\""
Jul 16 12:33:40.889772 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:40.889529    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a96583\"/\"default-dockercfg-jf45g\""
Jul 16 12:33:40.889772 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:40.889543    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a96583\"/\"auth-test-model-a96583-kserve-self-signed-certs\""
Jul 16 12:33:40.897942 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:40.897919    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"]
Jul 16 12:33:41.030321 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.030292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-home\") pod \"auth-test-model-a96583-kserve-6bf6bb49c9-nctd4\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") " pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:41.032187 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.030334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-dshm\") pod \"auth-test-model-a96583-kserve-6bf6bb49c9-nctd4\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") " pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:41.032187 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.030434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f225f84e-b6f7-41b6-b842-d891a2d83278-tls-certs\") pod \"auth-test-model-a96583-kserve-6bf6bb49c9-nctd4\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") " pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:41.032187 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.030486    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-tmp-dir\") pod \"auth-test-model-a96583-kserve-6bf6bb49c9-nctd4\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") " pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:41.032187 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.030524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-model-cache\") pod \"auth-test-model-a96583-kserve-6bf6bb49c9-nctd4\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") " pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:41.032187 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.030548    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkcxx\" (UniqueName: \"kubernetes.io/projected/f225f84e-b6f7-41b6-b842-d891a2d83278-kube-api-access-qkcxx\") pod \"auth-test-model-a96583-kserve-6bf6bb49c9-nctd4\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") " pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:41.117707 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.117681    2578 patch_prober.go:28] interesting pod/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 16 12:33:41.117814 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.117735    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl" podUID="f45abf0a-af01-4622-8da8-ebd7d79c1bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 16 12:33:41.131092 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.131064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-tmp-dir\") pod \"auth-test-model-a96583-kserve-6bf6bb49c9-nctd4\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") " pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:41.131211 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.131105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-model-cache\") pod \"auth-test-model-a96583-kserve-6bf6bb49c9-nctd4\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") " pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:41.131211 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.131130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkcxx\" (UniqueName: \"kubernetes.io/projected/f225f84e-b6f7-41b6-b842-d891a2d83278-kube-api-access-qkcxx\") pod \"auth-test-model-a96583-kserve-6bf6bb49c9-nctd4\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") " pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:41.131211 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.131177    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-home\") pod \"auth-test-model-a96583-kserve-6bf6bb49c9-nctd4\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") " pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:41.131211 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.131200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-dshm\") pod \"auth-test-model-a96583-kserve-6bf6bb49c9-nctd4\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") " pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:41.131427 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.131231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f225f84e-b6f7-41b6-b842-d891a2d83278-tls-certs\") pod \"auth-test-model-a96583-kserve-6bf6bb49c9-nctd4\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") " pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:41.131483 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.131442    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-tmp-dir\") pod \"auth-test-model-a96583-kserve-6bf6bb49c9-nctd4\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") " pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:41.131547 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.131519    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-model-cache\") pod \"auth-test-model-a96583-kserve-6bf6bb49c9-nctd4\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") " pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:41.131587 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.131534    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-home\") pod \"auth-test-model-a96583-kserve-6bf6bb49c9-nctd4\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") " pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:41.133394 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.133351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-dshm\") pod \"auth-test-model-a96583-kserve-6bf6bb49c9-nctd4\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") " pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:41.133548 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.133534    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f225f84e-b6f7-41b6-b842-d891a2d83278-tls-certs\") pod \"auth-test-model-a96583-kserve-6bf6bb49c9-nctd4\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") " pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:41.139305 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.139282    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkcxx\" (UniqueName: \"kubernetes.io/projected/f225f84e-b6f7-41b6-b842-d891a2d83278-kube-api-access-qkcxx\") pod \"auth-test-model-a96583-kserve-6bf6bb49c9-nctd4\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") " pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:41.197309 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.197282    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:41.205072 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.205027    2578 generic.go:356] "Generic (PLEG): container finished" podID="f45abf0a-af01-4622-8da8-ebd7d79c1bca" containerID="e2e7741462a31973a645893587a154f636a20bedf6374728a36bb98a8be2d259" exitCode=0
Jul 16 12:33:41.205173 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.205106    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl" event={"ID":"f45abf0a-af01-4622-8da8-ebd7d79c1bca","Type":"ContainerDied","Data":"e2e7741462a31973a645893587a154f636a20bedf6374728a36bb98a8be2d259"}
Jul 16 12:33:41.205559 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.205542    2578 scope.go:117] "RemoveContainer" containerID="e2e7741462a31973a645893587a154f636a20bedf6374728a36bb98a8be2d259"
Jul 16 12:33:41.207561 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.207531    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:33:41.207639 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.207618    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:33:41.207678 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.207660    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 12:33:41.359015 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:41.358962    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"]
Jul 16 12:33:41.362315 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:33:41.362291    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf225f84e_b6f7_41b6_b842_d891a2d83278.slice/crio-95ab1c23daa4f153397041c95011baf22dc2b1d8828bda424aa4f4c9748e513b WatchSource:0}: Error finding container 95ab1c23daa4f153397041c95011baf22dc2b1d8828bda424aa4f4c9748e513b: Status 404 returned error can't find the container with id 95ab1c23daa4f153397041c95011baf22dc2b1d8828bda424aa4f4c9748e513b
Jul 16 12:33:42.211607 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:42.211560    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4" event={"ID":"f225f84e-b6f7-41b6-b842-d891a2d83278","Type":"ContainerStarted","Data":"49949519d34a53b973f0da56b87eb88b43fc9d01fcc87a13955a06152ee3abce"}
Jul 16 12:33:42.212057 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:42.211621    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4" event={"ID":"f225f84e-b6f7-41b6-b842-d891a2d83278","Type":"ContainerStarted","Data":"95ab1c23daa4f153397041c95011baf22dc2b1d8828bda424aa4f4c9748e513b"}
Jul 16 12:33:42.213385 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:42.213363    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl" event={"ID":"f45abf0a-af01-4622-8da8-ebd7d79c1bca","Type":"ContainerStarted","Data":"61a26169803ef65069f73a3ed453e65a92e01ad369f8f0c14d3e26db116400f2"}
Jul 16 12:33:42.233649 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:42.233584    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4" podStartSLOduration=2.23354833 podStartE2EDuration="2.23354833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:33:42.231346933 +0000 UTC m=+2563.532124032" watchObservedRunningTime="2026-07-16 12:33:42.23354833 +0000 UTC m=+2563.534325431"
Jul 16 12:33:42.763320 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:42.763286    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:42.763603 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:42.763576    2578 patch_prober.go:28] interesting pod/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 16 12:33:42.763659 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:42.763635    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl" podUID="f45abf0a-af01-4622-8da8-ebd7d79c1bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 16 12:33:43.763612 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:43.763571    2578 patch_prober.go:28] interesting pod/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 16 12:33:43.764090 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:43.763632    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl" podUID="f45abf0a-af01-4622-8da8-ebd7d79c1bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 16 12:33:44.762787 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:44.762750    2578 patch_prober.go:28] interesting pod/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 16 12:33:44.763000 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:44.762804    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl" podUID="f45abf0a-af01-4622-8da8-ebd7d79c1bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 16 12:33:45.763466 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:45.763430    2578 patch_prober.go:28] interesting pod/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 16 12:33:45.763948 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:45.763480    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl" podUID="f45abf0a-af01-4622-8da8-ebd7d79c1bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 16 12:33:46.767995 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:46.767965    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:46.768408 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:46.768256    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:46.769112 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:46.769093    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:33:51.197905 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:51.197873    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:51.198360 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:51.198143    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:51.210409 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:51.210382    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:33:51.274330 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:33:51.274308    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:34:07.822817 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.822785    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"]
Jul 16 12:34:07.826592 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.826568    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:07.854538 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.854505    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-66e9a3\"/\"openshift-service-ca.crt\""
Jul 16 12:34:07.854900 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.854884    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-66e9a3\"/\"default-dockercfg-vcxsr\""
Jul 16 12:34:07.855003 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.854904    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-66e9a3\"/\"kube-root-ca.crt\""
Jul 16 12:34:07.855436 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.855414    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-66e9a3\"/\"auth-test-model-66e9a3-kserve-self-signed-certs\""
Jul 16 12:34:07.857015 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.856991    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"]
Jul 16 12:34:07.886698 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.886634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-home\") pod \"auth-test-model-66e9a3-kserve-67694c5595-xlhvx\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") " pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:07.886835 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.886767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-model-cache\") pod \"auth-test-model-66e9a3-kserve-67694c5595-xlhvx\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") " pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:07.886835 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.886809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-dshm\") pod \"auth-test-model-66e9a3-kserve-67694c5595-xlhvx\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") " pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:07.887008 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.886880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46bgr\" (UniqueName: \"kubernetes.io/projected/01052f8c-5fdb-435b-8291-e2a61104defd-kube-api-access-46bgr\") pod \"auth-test-model-66e9a3-kserve-67694c5595-xlhvx\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") " pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:07.887008 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.886945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-tmp-dir\") pod \"auth-test-model-66e9a3-kserve-67694c5595-xlhvx\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") " pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:07.887008 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.886962    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01052f8c-5fdb-435b-8291-e2a61104defd-tls-certs\") pod \"auth-test-model-66e9a3-kserve-67694c5595-xlhvx\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") " pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:07.988201 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.988163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46bgr\" (UniqueName: \"kubernetes.io/projected/01052f8c-5fdb-435b-8291-e2a61104defd-kube-api-access-46bgr\") pod \"auth-test-model-66e9a3-kserve-67694c5595-xlhvx\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") " pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:07.988392 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.988220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-tmp-dir\") pod \"auth-test-model-66e9a3-kserve-67694c5595-xlhvx\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") " pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:07.988392 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.988237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01052f8c-5fdb-435b-8291-e2a61104defd-tls-certs\") pod \"auth-test-model-66e9a3-kserve-67694c5595-xlhvx\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") " pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:07.988392 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.988271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-home\") pod \"auth-test-model-66e9a3-kserve-67694c5595-xlhvx\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") " pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:07.988392 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.988316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-model-cache\") pod \"auth-test-model-66e9a3-kserve-67694c5595-xlhvx\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") " pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:07.988392 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.988346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-dshm\") pod \"auth-test-model-66e9a3-kserve-67694c5595-xlhvx\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") " pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:07.988700 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.988676    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-tmp-dir\") pod \"auth-test-model-66e9a3-kserve-67694c5595-xlhvx\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") " pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:07.988773 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.988710    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-home\") pod \"auth-test-model-66e9a3-kserve-67694c5595-xlhvx\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") " pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:07.988773 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.988728    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-model-cache\") pod \"auth-test-model-66e9a3-kserve-67694c5595-xlhvx\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") " pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:07.990551 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.990523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-dshm\") pod \"auth-test-model-66e9a3-kserve-67694c5595-xlhvx\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") " pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:07.990804 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.990787    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01052f8c-5fdb-435b-8291-e2a61104defd-tls-certs\") pod \"auth-test-model-66e9a3-kserve-67694c5595-xlhvx\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") " pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:07.997412 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:07.997387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46bgr\" (UniqueName: \"kubernetes.io/projected/01052f8c-5fdb-435b-8291-e2a61104defd-kube-api-access-46bgr\") pod \"auth-test-model-66e9a3-kserve-67694c5595-xlhvx\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") " pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:08.137789 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:08.137702    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:08.281396 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:08.281371    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"]
Jul 16 12:34:08.283596 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:34:08.283563    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01052f8c_5fdb_435b_8291_e2a61104defd.slice/crio-78fae6d834a822c11df526c607375685e8b916633578cab17eda184261e6fc40 WatchSource:0}: Error finding container 78fae6d834a822c11df526c607375685e8b916633578cab17eda184261e6fc40: Status 404 returned error can't find the container with id 78fae6d834a822c11df526c607375685e8b916633578cab17eda184261e6fc40
Jul 16 12:34:08.347194 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:08.347153    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx" event={"ID":"01052f8c-5fdb-435b-8291-e2a61104defd","Type":"ContainerStarted","Data":"78fae6d834a822c11df526c607375685e8b916633578cab17eda184261e6fc40"}
Jul 16 12:34:09.353794 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:09.353757    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx" event={"ID":"01052f8c-5fdb-435b-8291-e2a61104defd","Type":"ContainerStarted","Data":"0776731035f915631360c40f09ae0deb54337374419d727630923f98c96c61d6"}
Jul 16 12:34:09.373697 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:09.373645    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx" podStartSLOduration=2.373630823 podStartE2EDuration="2.373630823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:34:09.373081159 +0000 UTC m=+2590.673858259" watchObservedRunningTime="2026-07-16 12:34:09.373630823 +0000 UTC m=+2590.674407921"
Jul 16 12:34:18.137832 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:18.137797    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:18.137832 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:18.137848    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:18.150478 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:18.150454    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:34:18.409508 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:34:18.409422    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:36:04.893200 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:04.893165    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"]
Jul 16 12:36:05.926595 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:05.926562    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"]
Jul 16 12:36:19.913284 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:19.913225    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4" podUID="f225f84e-b6f7-41b6-b842-d891a2d83278" containerName="main" containerID="cri-o://49949519d34a53b973f0da56b87eb88b43fc9d01fcc87a13955a06152ee3abce" gracePeriod=45
Jul 16 12:36:20.164169 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:20.164113    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:36:20.216672 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:20.216635    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-tmp-dir\") pod \"f225f84e-b6f7-41b6-b842-d891a2d83278\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") "
Jul 16 12:36:20.216824 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:20.216687    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-home\") pod \"f225f84e-b6f7-41b6-b842-d891a2d83278\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") "
Jul 16 12:36:20.216824 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:20.216715    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-dshm\") pod \"f225f84e-b6f7-41b6-b842-d891a2d83278\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") "
Jul 16 12:36:20.216824 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:20.216745    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkcxx\" (UniqueName: \"kubernetes.io/projected/f225f84e-b6f7-41b6-b842-d891a2d83278-kube-api-access-qkcxx\") pod \"f225f84e-b6f7-41b6-b842-d891a2d83278\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") "
Jul 16 12:36:20.216824 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:20.216790    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-model-cache\") pod \"f225f84e-b6f7-41b6-b842-d891a2d83278\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") "
Jul 16 12:36:20.217061 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:20.216904    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f225f84e-b6f7-41b6-b842-d891a2d83278" (UID: "f225f84e-b6f7-41b6-b842-d891a2d83278"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:36:20.217061 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:20.216920    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-home" (OuterVolumeSpecName: "home") pod "f225f84e-b6f7-41b6-b842-d891a2d83278" (UID: "f225f84e-b6f7-41b6-b842-d891a2d83278"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:36:20.217061 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:20.216977    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f225f84e-b6f7-41b6-b842-d891a2d83278-tls-certs\") pod \"f225f84e-b6f7-41b6-b842-d891a2d83278\" (UID: \"f225f84e-b6f7-41b6-b842-d891a2d83278\") "
Jul 16 12:36:20.217223 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:20.217123    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-model-cache" (OuterVolumeSpecName: "model-cache") pod "f225f84e-b6f7-41b6-b842-d891a2d83278" (UID: "f225f84e-b6f7-41b6-b842-d891a2d83278"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:36:20.217350 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:20.217332    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:36:20.217406 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:20.217352    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-tmp-dir\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:36:20.217406 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:20.217360    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-home\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:36:20.219167 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:20.219135    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f225f84e-b6f7-41b6-b842-d891a2d83278-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f225f84e-b6f7-41b6-b842-d891a2d83278" (UID: "f225f84e-b6f7-41b6-b842-d891a2d83278"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 12:36:20.219291 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:20.219235    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-dshm" (OuterVolumeSpecName: "dshm") pod "f225f84e-b6f7-41b6-b842-d891a2d83278" (UID: "f225f84e-b6f7-41b6-b842-d891a2d83278"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:36:20.219291 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:20.219278    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f225f84e-b6f7-41b6-b842-d891a2d83278-kube-api-access-qkcxx" (OuterVolumeSpecName: "kube-api-access-qkcxx") pod "f225f84e-b6f7-41b6-b842-d891a2d83278" (UID: "f225f84e-b6f7-41b6-b842-d891a2d83278"). InnerVolumeSpecName "kube-api-access-qkcxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:36:20.318437 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:20.318408    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f225f84e-b6f7-41b6-b842-d891a2d83278-dshm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:36:20.318437 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:20.318431    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkcxx\" (UniqueName: \"kubernetes.io/projected/f225f84e-b6f7-41b6-b842-d891a2d83278-kube-api-access-qkcxx\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:36:20.318437 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:20.318441    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f225f84e-b6f7-41b6-b842-d891a2d83278-tls-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:36:20.947108 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:20.947070    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx" podUID="01052f8c-5fdb-435b-8291-e2a61104defd" containerName="main" containerID="cri-o://0776731035f915631360c40f09ae0deb54337374419d727630923f98c96c61d6" gracePeriod=45
Jul 16 12:36:21.007881 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.007847    2578 generic.go:356] "Generic (PLEG): container finished" podID="f225f84e-b6f7-41b6-b842-d891a2d83278" containerID="49949519d34a53b973f0da56b87eb88b43fc9d01fcc87a13955a06152ee3abce" exitCode=0
Jul 16 12:36:21.008057 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.007891    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"
Jul 16 12:36:21.008057 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.007930    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4" event={"ID":"f225f84e-b6f7-41b6-b842-d891a2d83278","Type":"ContainerDied","Data":"49949519d34a53b973f0da56b87eb88b43fc9d01fcc87a13955a06152ee3abce"}
Jul 16 12:36:21.008057 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.007976    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4" event={"ID":"f225f84e-b6f7-41b6-b842-d891a2d83278","Type":"ContainerDied","Data":"95ab1c23daa4f153397041c95011baf22dc2b1d8828bda424aa4f4c9748e513b"}
Jul 16 12:36:21.008057 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.007994    2578 scope.go:117] "RemoveContainer" containerID="49949519d34a53b973f0da56b87eb88b43fc9d01fcc87a13955a06152ee3abce"
Jul 16 12:36:21.040033 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.040008    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"]
Jul 16 12:36:21.046233 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.046209    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a96583/auth-test-model-a96583-kserve-6bf6bb49c9-nctd4"]
Jul 16 12:36:21.057816 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.057799    2578 scope.go:117] "RemoveContainer" containerID="49949519d34a53b973f0da56b87eb88b43fc9d01fcc87a13955a06152ee3abce"
Jul 16 12:36:21.058122 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:36:21.058100    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49949519d34a53b973f0da56b87eb88b43fc9d01fcc87a13955a06152ee3abce\": container with ID starting with 49949519d34a53b973f0da56b87eb88b43fc9d01fcc87a13955a06152ee3abce not found: ID does not exist" containerID="49949519d34a53b973f0da56b87eb88b43fc9d01fcc87a13955a06152ee3abce"
Jul 16 12:36:21.058211 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.058129    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49949519d34a53b973f0da56b87eb88b43fc9d01fcc87a13955a06152ee3abce"} err="failed to get container status \"49949519d34a53b973f0da56b87eb88b43fc9d01fcc87a13955a06152ee3abce\": rpc error: code = NotFound desc = could not find container \"49949519d34a53b973f0da56b87eb88b43fc9d01fcc87a13955a06152ee3abce\": container with ID starting with 49949519d34a53b973f0da56b87eb88b43fc9d01fcc87a13955a06152ee3abce not found: ID does not exist"
Jul 16 12:36:21.206623 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.206561    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:36:21.228215 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.228190    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-tmp-dir\") pod \"01052f8c-5fdb-435b-8291-e2a61104defd\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") "
Jul 16 12:36:21.228357 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.228292    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-home\") pod \"01052f8c-5fdb-435b-8291-e2a61104defd\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") "
Jul 16 12:36:21.228420 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.228352    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-model-cache\") pod \"01052f8c-5fdb-435b-8291-e2a61104defd\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") "
Jul 16 12:36:21.228420 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.228381    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01052f8c-5fdb-435b-8291-e2a61104defd-tls-certs\") pod \"01052f8c-5fdb-435b-8291-e2a61104defd\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") "
Jul 16 12:36:21.228531 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.228420    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46bgr\" (UniqueName: \"kubernetes.io/projected/01052f8c-5fdb-435b-8291-e2a61104defd-kube-api-access-46bgr\") pod \"01052f8c-5fdb-435b-8291-e2a61104defd\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") "
Jul 16 12:36:21.228531 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.228447    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-dshm\") pod \"01052f8c-5fdb-435b-8291-e2a61104defd\" (UID: \"01052f8c-5fdb-435b-8291-e2a61104defd\") "
Jul 16 12:36:21.228531 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.228454    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "01052f8c-5fdb-435b-8291-e2a61104defd" (UID: "01052f8c-5fdb-435b-8291-e2a61104defd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:36:21.228686 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.228589    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-home" (OuterVolumeSpecName: "home") pod "01052f8c-5fdb-435b-8291-e2a61104defd" (UID: "01052f8c-5fdb-435b-8291-e2a61104defd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:36:21.228804 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.228784    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-home\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:36:21.228862 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.228818    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-tmp-dir\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:36:21.228862 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.228789    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-model-cache" (OuterVolumeSpecName: "model-cache") pod "01052f8c-5fdb-435b-8291-e2a61104defd" (UID: "01052f8c-5fdb-435b-8291-e2a61104defd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:36:21.231597 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.231526    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-dshm" (OuterVolumeSpecName: "dshm") pod "01052f8c-5fdb-435b-8291-e2a61104defd" (UID: "01052f8c-5fdb-435b-8291-e2a61104defd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:36:21.231797 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.231678    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01052f8c-5fdb-435b-8291-e2a61104defd-kube-api-access-46bgr" (OuterVolumeSpecName: "kube-api-access-46bgr") pod "01052f8c-5fdb-435b-8291-e2a61104defd" (UID: "01052f8c-5fdb-435b-8291-e2a61104defd"). InnerVolumeSpecName "kube-api-access-46bgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:36:21.231797 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.231705    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01052f8c-5fdb-435b-8291-e2a61104defd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01052f8c-5fdb-435b-8291-e2a61104defd" (UID: "01052f8c-5fdb-435b-8291-e2a61104defd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 12:36:21.300136 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.300107    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f225f84e-b6f7-41b6-b842-d891a2d83278" path="/var/lib/kubelet/pods/f225f84e-b6f7-41b6-b842-d891a2d83278/volumes"
Jul 16 12:36:21.329379 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.329346    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:36:21.329379 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.329370    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01052f8c-5fdb-435b-8291-e2a61104defd-tls-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:36:21.329557 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.329383    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46bgr\" (UniqueName: \"kubernetes.io/projected/01052f8c-5fdb-435b-8291-e2a61104defd-kube-api-access-46bgr\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:36:21.329557 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:21.329469    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01052f8c-5fdb-435b-8291-e2a61104defd-dshm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:36:22.014678 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:22.014643    2578 generic.go:356] "Generic (PLEG): container finished" podID="01052f8c-5fdb-435b-8291-e2a61104defd" containerID="0776731035f915631360c40f09ae0deb54337374419d727630923f98c96c61d6" exitCode=0
Jul 16 12:36:22.014678 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:22.014680    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"
Jul 16 12:36:22.015195 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:22.014682    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx" event={"ID":"01052f8c-5fdb-435b-8291-e2a61104defd","Type":"ContainerDied","Data":"0776731035f915631360c40f09ae0deb54337374419d727630923f98c96c61d6"}
Jul 16 12:36:22.015195 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:22.014779    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx" event={"ID":"01052f8c-5fdb-435b-8291-e2a61104defd","Type":"ContainerDied","Data":"78fae6d834a822c11df526c607375685e8b916633578cab17eda184261e6fc40"}
Jul 16 12:36:22.015195 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:22.014798    2578 scope.go:117] "RemoveContainer" containerID="0776731035f915631360c40f09ae0deb54337374419d727630923f98c96c61d6"
Jul 16 12:36:22.022975 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:22.022958    2578 scope.go:117] "RemoveContainer" containerID="0776731035f915631360c40f09ae0deb54337374419d727630923f98c96c61d6"
Jul 16 12:36:22.023268 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:36:22.023246    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0776731035f915631360c40f09ae0deb54337374419d727630923f98c96c61d6\": container with ID starting with 0776731035f915631360c40f09ae0deb54337374419d727630923f98c96c61d6 not found: ID does not exist" containerID="0776731035f915631360c40f09ae0deb54337374419d727630923f98c96c61d6"
Jul 16 12:36:22.023375 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:22.023274    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0776731035f915631360c40f09ae0deb54337374419d727630923f98c96c61d6"} err="failed to get container status \"0776731035f915631360c40f09ae0deb54337374419d727630923f98c96c61d6\": rpc error: code = NotFound desc = could not find container \"0776731035f915631360c40f09ae0deb54337374419d727630923f98c96c61d6\": container with ID starting with 0776731035f915631360c40f09ae0deb54337374419d727630923f98c96c61d6 not found: ID does not exist"
Jul 16 12:36:22.037182 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:22.037157    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"]
Jul 16 12:36:22.045291 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:22.045270    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-66e9a3/auth-test-model-66e9a3-kserve-67694c5595-xlhvx"]
Jul 16 12:36:23.300002 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:23.299970    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01052f8c-5fdb-435b-8291-e2a61104defd" path="/var/lib/kubelet/pods/01052f8c-5fdb-435b-8291-e2a61104defd/volumes"
Jul 16 12:36:25.008247 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.008216    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"]
Jul 16 12:36:25.008793 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.008774    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01052f8c-5fdb-435b-8291-e2a61104defd" containerName="main"
Jul 16 12:36:25.008877 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.008797    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="01052f8c-5fdb-435b-8291-e2a61104defd" containerName="main"
Jul 16 12:36:25.008877 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.008819    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f225f84e-b6f7-41b6-b842-d891a2d83278" containerName="main"
Jul 16 12:36:25.008877 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.008828    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f225f84e-b6f7-41b6-b842-d891a2d83278" containerName="main"
Jul 16 12:36:25.009032 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.008943    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="01052f8c-5fdb-435b-8291-e2a61104defd" containerName="main"
Jul 16 12:36:25.009032 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.008961    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f225f84e-b6f7-41b6-b842-d891a2d83278" containerName="main"
Jul 16 12:36:25.013217 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.013194    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:25.017692 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.017672    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a96583\"/\"openshift-service-ca.crt\""
Jul 16 12:36:25.017692 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.017683    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a96583\"/\"sub-test-model-a96583-kserve-self-signed-certs\""
Jul 16 12:36:25.017875 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.017694    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a96583\"/\"default-dockercfg-jf45g\""
Jul 16 12:36:25.017875 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.017708    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a96583\"/\"kube-root-ca.crt\""
Jul 16 12:36:25.033258 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.033212    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"]
Jul 16 12:36:25.063983 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.063946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e79d86-e288-4d23-829a-4b2108fe9b6e-tls-certs\") pod \"sub-test-model-a96583-kserve-74c588988c-6f27f\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") " pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:25.063983 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.063987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-model-cache\") pod \"sub-test-model-a96583-kserve-74c588988c-6f27f\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") " pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:25.064195 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.064071    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-home\") pod \"sub-test-model-a96583-kserve-74c588988c-6f27f\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") " pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:25.064195 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.064096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-dshm\") pod \"sub-test-model-a96583-kserve-74c588988c-6f27f\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") " pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:25.064195 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.064120    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs95b\" (UniqueName: \"kubernetes.io/projected/a7e79d86-e288-4d23-829a-4b2108fe9b6e-kube-api-access-qs95b\") pod \"sub-test-model-a96583-kserve-74c588988c-6f27f\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") " pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:25.064195 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.064164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-tmp-dir\") pod \"sub-test-model-a96583-kserve-74c588988c-6f27f\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") " pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:25.164871 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.164835    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-tmp-dir\") pod \"sub-test-model-a96583-kserve-74c588988c-6f27f\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") " pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:25.165073 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.164898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e79d86-e288-4d23-829a-4b2108fe9b6e-tls-certs\") pod \"sub-test-model-a96583-kserve-74c588988c-6f27f\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") " pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:25.165073 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.164928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-model-cache\") pod \"sub-test-model-a96583-kserve-74c588988c-6f27f\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") " pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:25.165073 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.164991    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-home\") pod \"sub-test-model-a96583-kserve-74c588988c-6f27f\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") " pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:25.165073 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.165014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-dshm\") pod \"sub-test-model-a96583-kserve-74c588988c-6f27f\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") " pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:25.165073 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.165058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs95b\" (UniqueName: \"kubernetes.io/projected/a7e79d86-e288-4d23-829a-4b2108fe9b6e-kube-api-access-qs95b\") pod \"sub-test-model-a96583-kserve-74c588988c-6f27f\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") " pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:25.165370 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.165348    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-tmp-dir\") pod \"sub-test-model-a96583-kserve-74c588988c-6f27f\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") " pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:25.165440 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.165382    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-model-cache\") pod \"sub-test-model-a96583-kserve-74c588988c-6f27f\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") " pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:25.165502 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.165465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-home\") pod \"sub-test-model-a96583-kserve-74c588988c-6f27f\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") " pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:25.167527 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.167502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e79d86-e288-4d23-829a-4b2108fe9b6e-tls-certs\") pod \"sub-test-model-a96583-kserve-74c588988c-6f27f\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") " pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:25.167527 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.167514    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-dshm\") pod \"sub-test-model-a96583-kserve-74c588988c-6f27f\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") " pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:25.173848 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.173822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs95b\" (UniqueName: \"kubernetes.io/projected/a7e79d86-e288-4d23-829a-4b2108fe9b6e-kube-api-access-qs95b\") pod \"sub-test-model-a96583-kserve-74c588988c-6f27f\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") " pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:25.325633 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.325600    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:25.460012 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:25.459988    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"]
Jul 16 12:36:25.462497 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:36:25.462461    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7e79d86_e288_4d23_829a_4b2108fe9b6e.slice/crio-c2bdbcb1c3a320ef496e65e0cddaf9621f51ab61b786c6a1633f0100b0fd4277 WatchSource:0}: Error finding container c2bdbcb1c3a320ef496e65e0cddaf9621f51ab61b786c6a1633f0100b0fd4277: Status 404 returned error can't find the container with id c2bdbcb1c3a320ef496e65e0cddaf9621f51ab61b786c6a1633f0100b0fd4277
Jul 16 12:36:26.041393 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:26.041358    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f" event={"ID":"a7e79d86-e288-4d23-829a-4b2108fe9b6e","Type":"ContainerStarted","Data":"7de041d91e1cb95fa22d6a55f77e8ff25aeab0a70e3112a824225508822d7f8b"}
Jul 16 12:36:26.041763 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:26.041408    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f" event={"ID":"a7e79d86-e288-4d23-829a-4b2108fe9b6e","Type":"ContainerStarted","Data":"c2bdbcb1c3a320ef496e65e0cddaf9621f51ab61b786c6a1633f0100b0fd4277"}
Jul 16 12:36:26.061497 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:26.061444    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f" podStartSLOduration=2.061428245 podStartE2EDuration="2.061428245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:36:26.059400218 +0000 UTC m=+2727.360177315" watchObservedRunningTime="2026-07-16 12:36:26.061428245 +0000 UTC m=+2727.362205343"
Jul 16 12:36:35.325876 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:35.325846    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:35.326290 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:35.325895    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:35.338916 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:35.338887    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:36:36.100619 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:36:36.100589    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:37:40.078510 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:40.078434    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"]
Jul 16 12:37:45.798430 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:45.798393    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"]
Jul 16 12:37:45.801912 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:45.801892    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:45.805076 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:45.805048    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a96583\"/\"rate-test-model-a96583-kserve-self-signed-certs\""
Jul 16 12:37:45.817357 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:45.817322    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"]
Jul 16 12:37:45.909711 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:45.909672    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5txw5\" (UniqueName: \"kubernetes.io/projected/ce2790a8-08d9-4dfb-baad-87abb63401f5-kube-api-access-5txw5\") pod \"rate-test-model-a96583-kserve-6dc6f5f64c-4s424\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") " pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:45.909711 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:45.909715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-home\") pod \"rate-test-model-a96583-kserve-6dc6f5f64c-4s424\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") " pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:45.909932 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:45.909752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-dshm\") pod \"rate-test-model-a96583-kserve-6dc6f5f64c-4s424\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") " pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:45.909932 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:45.909771    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-tmp-dir\") pod \"rate-test-model-a96583-kserve-6dc6f5f64c-4s424\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") " pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:45.909932 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:45.909819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce2790a8-08d9-4dfb-baad-87abb63401f5-tls-certs\") pod \"rate-test-model-a96583-kserve-6dc6f5f64c-4s424\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") " pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:45.909932 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:45.909848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-model-cache\") pod \"rate-test-model-a96583-kserve-6dc6f5f64c-4s424\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") " pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:46.010821 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:46.010780    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5txw5\" (UniqueName: \"kubernetes.io/projected/ce2790a8-08d9-4dfb-baad-87abb63401f5-kube-api-access-5txw5\") pod \"rate-test-model-a96583-kserve-6dc6f5f64c-4s424\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") " pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:46.011026 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:46.010826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-home\") pod \"rate-test-model-a96583-kserve-6dc6f5f64c-4s424\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") " pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:46.011026 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:46.010869    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-dshm\") pod \"rate-test-model-a96583-kserve-6dc6f5f64c-4s424\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") " pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:46.011026 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:46.010890    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-tmp-dir\") pod \"rate-test-model-a96583-kserve-6dc6f5f64c-4s424\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") " pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:46.011026 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:46.010957    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce2790a8-08d9-4dfb-baad-87abb63401f5-tls-certs\") pod \"rate-test-model-a96583-kserve-6dc6f5f64c-4s424\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") " pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:46.011026 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:46.011003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-model-cache\") pod \"rate-test-model-a96583-kserve-6dc6f5f64c-4s424\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") " pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:46.011379 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:46.011346    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-home\") pod \"rate-test-model-a96583-kserve-6dc6f5f64c-4s424\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") " pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:46.011446 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:46.011387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-tmp-dir\") pod \"rate-test-model-a96583-kserve-6dc6f5f64c-4s424\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") " pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:46.011507 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:46.011468    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-model-cache\") pod \"rate-test-model-a96583-kserve-6dc6f5f64c-4s424\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") " pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:46.013217 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:46.013195    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-dshm\") pod \"rate-test-model-a96583-kserve-6dc6f5f64c-4s424\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") " pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:46.013602 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:46.013579    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce2790a8-08d9-4dfb-baad-87abb63401f5-tls-certs\") pod \"rate-test-model-a96583-kserve-6dc6f5f64c-4s424\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") " pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:46.032012 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:46.031979    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5txw5\" (UniqueName: \"kubernetes.io/projected/ce2790a8-08d9-4dfb-baad-87abb63401f5-kube-api-access-5txw5\") pod \"rate-test-model-a96583-kserve-6dc6f5f64c-4s424\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") " pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:46.111492 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:46.111402    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:46.253861 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:46.253832    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"]
Jul 16 12:37:46.255313 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:37:46.255279    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce2790a8_08d9_4dfb_baad_87abb63401f5.slice/crio-3f79af2d08ebd077492e89a94379c36ed8b6752e45d1755fe0b7b6e743479dff WatchSource:0}: Error finding container 3f79af2d08ebd077492e89a94379c36ed8b6752e45d1755fe0b7b6e743479dff: Status 404 returned error can't find the container with id 3f79af2d08ebd077492e89a94379c36ed8b6752e45d1755fe0b7b6e743479dff
Jul 16 12:37:46.431822 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:46.431728    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424" event={"ID":"ce2790a8-08d9-4dfb-baad-87abb63401f5","Type":"ContainerStarted","Data":"953e9f89d69775adae06e606da32911b2c9702b26c55f7fc1094021e5d9213b6"}
Jul 16 12:37:46.431822 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:46.431788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424" event={"ID":"ce2790a8-08d9-4dfb-baad-87abb63401f5","Type":"ContainerStarted","Data":"3f79af2d08ebd077492e89a94379c36ed8b6752e45d1755fe0b7b6e743479dff"}
Jul 16 12:37:46.458291 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:46.458240    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424" podStartSLOduration=1.458222735 podStartE2EDuration="1.458222735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:37:46.457609196 +0000 UTC m=+2807.758386294" watchObservedRunningTime="2026-07-16 12:37:46.458222735 +0000 UTC m=+2807.758999832"
Jul 16 12:37:55.098525 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.098481    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f" podUID="a7e79d86-e288-4d23-829a-4b2108fe9b6e" containerName="main" containerID="cri-o://7de041d91e1cb95fa22d6a55f77e8ff25aeab0a70e3112a824225508822d7f8b" gracePeriod=45
Jul 16 12:37:55.361810 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.361754    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:37:55.406004 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.405957    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs95b\" (UniqueName: \"kubernetes.io/projected/a7e79d86-e288-4d23-829a-4b2108fe9b6e-kube-api-access-qs95b\") pod \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") "
Jul 16 12:37:55.406155 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.406025    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-model-cache\") pod \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") "
Jul 16 12:37:55.406155 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.406140    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-home\") pod \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") "
Jul 16 12:37:55.406260 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.406177    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e79d86-e288-4d23-829a-4b2108fe9b6e-tls-certs\") pod \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") "
Jul 16 12:37:55.406260 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.406206    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-dshm\") pod \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") "
Jul 16 12:37:55.406260 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.406218    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7e79d86-e288-4d23-829a-4b2108fe9b6e" (UID: "a7e79d86-e288-4d23-829a-4b2108fe9b6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:37:55.406260 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.406243    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-tmp-dir\") pod \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\" (UID: \"a7e79d86-e288-4d23-829a-4b2108fe9b6e\") "
Jul 16 12:37:55.406485 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.406430    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-home" (OuterVolumeSpecName: "home") pod "a7e79d86-e288-4d23-829a-4b2108fe9b6e" (UID: "a7e79d86-e288-4d23-829a-4b2108fe9b6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:37:55.406908 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.406643    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:37:55.406908 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.406666    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-home\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:37:55.406908 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.406861    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7e79d86-e288-4d23-829a-4b2108fe9b6e" (UID: "a7e79d86-e288-4d23-829a-4b2108fe9b6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:37:55.408295 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.408260    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e79d86-e288-4d23-829a-4b2108fe9b6e-kube-api-access-qs95b" (OuterVolumeSpecName: "kube-api-access-qs95b") pod "a7e79d86-e288-4d23-829a-4b2108fe9b6e" (UID: "a7e79d86-e288-4d23-829a-4b2108fe9b6e"). InnerVolumeSpecName "kube-api-access-qs95b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:37:55.408295 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.408281    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-dshm" (OuterVolumeSpecName: "dshm") pod "a7e79d86-e288-4d23-829a-4b2108fe9b6e" (UID: "a7e79d86-e288-4d23-829a-4b2108fe9b6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:37:55.408470 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.408312    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7e79d86-e288-4d23-829a-4b2108fe9b6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7e79d86-e288-4d23-829a-4b2108fe9b6e" (UID: "a7e79d86-e288-4d23-829a-4b2108fe9b6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 12:37:55.477947 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.477917    2578 generic.go:356] "Generic (PLEG): container finished" podID="a7e79d86-e288-4d23-829a-4b2108fe9b6e" containerID="7de041d91e1cb95fa22d6a55f77e8ff25aeab0a70e3112a824225508822d7f8b" exitCode=0
Jul 16 12:37:55.478114 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.477956    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"
Jul 16 12:37:55.478114 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.477965    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f" event={"ID":"a7e79d86-e288-4d23-829a-4b2108fe9b6e","Type":"ContainerDied","Data":"7de041d91e1cb95fa22d6a55f77e8ff25aeab0a70e3112a824225508822d7f8b"}
Jul 16 12:37:55.478114 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.477997    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f" event={"ID":"a7e79d86-e288-4d23-829a-4b2108fe9b6e","Type":"ContainerDied","Data":"c2bdbcb1c3a320ef496e65e0cddaf9621f51ab61b786c6a1633f0100b0fd4277"}
Jul 16 12:37:55.478114 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.478013    2578 scope.go:117] "RemoveContainer" containerID="7de041d91e1cb95fa22d6a55f77e8ff25aeab0a70e3112a824225508822d7f8b"
Jul 16 12:37:55.490216 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.488481    2578 scope.go:117] "RemoveContainer" containerID="7de041d91e1cb95fa22d6a55f77e8ff25aeab0a70e3112a824225508822d7f8b"
Jul 16 12:37:55.490216 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:37:55.488753    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7de041d91e1cb95fa22d6a55f77e8ff25aeab0a70e3112a824225508822d7f8b\": container with ID starting with 7de041d91e1cb95fa22d6a55f77e8ff25aeab0a70e3112a824225508822d7f8b not found: ID does not exist" containerID="7de041d91e1cb95fa22d6a55f77e8ff25aeab0a70e3112a824225508822d7f8b"
Jul 16 12:37:55.490216 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.488778    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de041d91e1cb95fa22d6a55f77e8ff25aeab0a70e3112a824225508822d7f8b"} err="failed to get container status \"7de041d91e1cb95fa22d6a55f77e8ff25aeab0a70e3112a824225508822d7f8b\": rpc error: code = NotFound desc = could not find container \"7de041d91e1cb95fa22d6a55f77e8ff25aeab0a70e3112a824225508822d7f8b\": container with ID starting with 7de041d91e1cb95fa22d6a55f77e8ff25aeab0a70e3112a824225508822d7f8b not found: ID does not exist"
Jul 16 12:37:55.505845 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.505815    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"]
Jul 16 12:37:55.507974 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.507958    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e79d86-e288-4d23-829a-4b2108fe9b6e-tls-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:37:55.508049 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.507976    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-dshm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:37:55.508049 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.507984    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e79d86-e288-4d23-829a-4b2108fe9b6e-tmp-dir\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:37:55.508049 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.507993    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs95b\" (UniqueName: \"kubernetes.io/projected/a7e79d86-e288-4d23-829a-4b2108fe9b6e-kube-api-access-qs95b\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:37:55.510888 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:55.510867    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a96583/sub-test-model-a96583-kserve-74c588988c-6f27f"]
Jul 16 12:37:56.112493 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:56.112458    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:56.112493 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:56.112503    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:56.125889 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:56.125857    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:56.495553 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:56.495479    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:37:57.299455 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:37:57.299423    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7e79d86-e288-4d23-829a-4b2108fe9b6e" path="/var/lib/kubelet/pods/a7e79d86-e288-4d23-829a-4b2108fe9b6e/volumes"
Jul 16 12:38:07.790280 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:07.790241    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"]
Jul 16 12:38:07.790646 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:07.790597    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7e79d86-e288-4d23-829a-4b2108fe9b6e" containerName="main"
Jul 16 12:38:07.790646 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:07.790607    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7e79d86-e288-4d23-829a-4b2108fe9b6e" containerName="main"
Jul 16 12:38:07.790719 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:07.790677    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7e79d86-e288-4d23-829a-4b2108fe9b6e" containerName="main"
Jul 16 12:38:07.795235 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:07.795209    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:07.798808 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:07.798771    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-66e9a3\"/\"default-dockercfg-vcxsr\""
Jul 16 12:38:07.798808 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:07.798804    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-66e9a3\"/\"rate-test-model-66e9a3-kserve-self-signed-certs\""
Jul 16 12:38:07.798989 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:07.798834    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-66e9a3\"/\"openshift-service-ca.crt\""
Jul 16 12:38:07.798989 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:07.798806    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-66e9a3\"/\"kube-root-ca.crt\""
Jul 16 12:38:07.804898 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:07.804876    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"]
Jul 16 12:38:07.921974 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:07.921938    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-tmp-dir\") pod \"rate-test-model-66e9a3-kserve-7b66d875ff-5mk82\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") " pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:07.922163 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:07.922006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njcjw\" (UniqueName: \"kubernetes.io/projected/ee02e34a-6edc-490c-a984-8af21bd69bcf-kube-api-access-njcjw\") pod \"rate-test-model-66e9a3-kserve-7b66d875ff-5mk82\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") " pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:07.922163 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:07.922024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-model-cache\") pod \"rate-test-model-66e9a3-kserve-7b66d875ff-5mk82\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") " pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:07.922163 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:07.922075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-dshm\") pod \"rate-test-model-66e9a3-kserve-7b66d875ff-5mk82\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") " pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:07.922163 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:07.922095    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-home\") pod \"rate-test-model-66e9a3-kserve-7b66d875ff-5mk82\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") " pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:07.922163 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:07.922126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee02e34a-6edc-490c-a984-8af21bd69bcf-tls-certs\") pod \"rate-test-model-66e9a3-kserve-7b66d875ff-5mk82\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") " pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:08.022847 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:08.022818    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-tmp-dir\") pod \"rate-test-model-66e9a3-kserve-7b66d875ff-5mk82\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") " pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:08.023033 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:08.022883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njcjw\" (UniqueName: \"kubernetes.io/projected/ee02e34a-6edc-490c-a984-8af21bd69bcf-kube-api-access-njcjw\") pod \"rate-test-model-66e9a3-kserve-7b66d875ff-5mk82\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") " pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:08.023033 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:08.022904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-model-cache\") pod \"rate-test-model-66e9a3-kserve-7b66d875ff-5mk82\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") " pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:08.023033 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:08.022926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-dshm\") pod \"rate-test-model-66e9a3-kserve-7b66d875ff-5mk82\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") " pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:08.023033 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:08.022943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-home\") pod \"rate-test-model-66e9a3-kserve-7b66d875ff-5mk82\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") " pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:08.023033 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:08.022977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee02e34a-6edc-490c-a984-8af21bd69bcf-tls-certs\") pod \"rate-test-model-66e9a3-kserve-7b66d875ff-5mk82\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") " pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:08.023348 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:08.023306    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-tmp-dir\") pod \"rate-test-model-66e9a3-kserve-7b66d875ff-5mk82\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") " pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:08.023348 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:08.023335    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-model-cache\") pod \"rate-test-model-66e9a3-kserve-7b66d875ff-5mk82\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") " pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:08.023565 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:08.023540    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-home\") pod \"rate-test-model-66e9a3-kserve-7b66d875ff-5mk82\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") " pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:08.025534 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:08.025504    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-dshm\") pod \"rate-test-model-66e9a3-kserve-7b66d875ff-5mk82\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") " pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:08.025939 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:08.025918    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee02e34a-6edc-490c-a984-8af21bd69bcf-tls-certs\") pod \"rate-test-model-66e9a3-kserve-7b66d875ff-5mk82\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") " pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:08.033925 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:08.033905    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njcjw\" (UniqueName: \"kubernetes.io/projected/ee02e34a-6edc-490c-a984-8af21bd69bcf-kube-api-access-njcjw\") pod \"rate-test-model-66e9a3-kserve-7b66d875ff-5mk82\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") " pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:08.104335 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:08.104307    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:08.248966 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:08.248936    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"]
Jul 16 12:38:08.250995 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:38:08.250972    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee02e34a_6edc_490c_a984_8af21bd69bcf.slice/crio-8ee8794f9f3d4f51f3069e163875611b525c43415009d2c1e7f8bef89826d23c WatchSource:0}: Error finding container 8ee8794f9f3d4f51f3069e163875611b525c43415009d2c1e7f8bef89826d23c: Status 404 returned error can't find the container with id 8ee8794f9f3d4f51f3069e163875611b525c43415009d2c1e7f8bef89826d23c
Jul 16 12:38:08.544378 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:08.544344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82" event={"ID":"ee02e34a-6edc-490c-a984-8af21bd69bcf","Type":"ContainerStarted","Data":"5cd586f2fcea9fbd5973e65f707e526f7d4563d66617f3ee6f7104bd0c091b4a"}
Jul 16 12:38:08.544536 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:08.544387    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82" event={"ID":"ee02e34a-6edc-490c-a984-8af21bd69bcf","Type":"ContainerStarted","Data":"8ee8794f9f3d4f51f3069e163875611b525c43415009d2c1e7f8bef89826d23c"}
Jul 16 12:38:18.105049 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:18.105012    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:18.105581 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:18.105095    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:18.117378 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:18.117344    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:18.139465 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:18.139416    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82" podStartSLOduration=11.139399876 podStartE2EDuration="11.139399876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:38:08.57096691 +0000 UTC m=+2829.871744007" watchObservedRunningTime="2026-07-16 12:38:18.139399876 +0000 UTC m=+2839.440176974"
Jul 16 12:38:18.610624 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:18.610596    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:38:55.365187 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:55.365105    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"]
Jul 16 12:38:56.586973 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:38:56.586942    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"]
Jul 16 12:39:10.384755 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.384716    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424" podUID="ce2790a8-08d9-4dfb-baad-87abb63401f5" containerName="main" containerID="cri-o://953e9f89d69775adae06e606da32911b2c9702b26c55f7fc1094021e5d9213b6" gracePeriod=45
Jul 16 12:39:10.647200 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.647136    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:39:10.710311 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.710278    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-model-cache\") pod \"ce2790a8-08d9-4dfb-baad-87abb63401f5\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") "
Jul 16 12:39:10.710494 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.710325    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-tmp-dir\") pod \"ce2790a8-08d9-4dfb-baad-87abb63401f5\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") "
Jul 16 12:39:10.710494 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.710397    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-dshm\") pod \"ce2790a8-08d9-4dfb-baad-87abb63401f5\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") "
Jul 16 12:39:10.710494 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.710417    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-home\") pod \"ce2790a8-08d9-4dfb-baad-87abb63401f5\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") "
Jul 16 12:39:10.710494 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.710447    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5txw5\" (UniqueName: \"kubernetes.io/projected/ce2790a8-08d9-4dfb-baad-87abb63401f5-kube-api-access-5txw5\") pod \"ce2790a8-08d9-4dfb-baad-87abb63401f5\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") "
Jul 16 12:39:10.710494 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.710476    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce2790a8-08d9-4dfb-baad-87abb63401f5-tls-certs\") pod \"ce2790a8-08d9-4dfb-baad-87abb63401f5\" (UID: \"ce2790a8-08d9-4dfb-baad-87abb63401f5\") "
Jul 16 12:39:10.710750 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.710558    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce2790a8-08d9-4dfb-baad-87abb63401f5" (UID: "ce2790a8-08d9-4dfb-baad-87abb63401f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:39:10.710750 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.710658    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-home" (OuterVolumeSpecName: "home") pod "ce2790a8-08d9-4dfb-baad-87abb63401f5" (UID: "ce2790a8-08d9-4dfb-baad-87abb63401f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:39:10.710750 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.710674    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce2790a8-08d9-4dfb-baad-87abb63401f5" (UID: "ce2790a8-08d9-4dfb-baad-87abb63401f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:39:10.710887 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.710805    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:10.710887 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.710819    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-tmp-dir\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:10.710887 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.710827    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-home\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:10.712611 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.712587    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-dshm" (OuterVolumeSpecName: "dshm") pod "ce2790a8-08d9-4dfb-baad-87abb63401f5" (UID: "ce2790a8-08d9-4dfb-baad-87abb63401f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:39:10.712611 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.712594    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce2790a8-08d9-4dfb-baad-87abb63401f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce2790a8-08d9-4dfb-baad-87abb63401f5" (UID: "ce2790a8-08d9-4dfb-baad-87abb63401f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 12:39:10.712783 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.712641    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2790a8-08d9-4dfb-baad-87abb63401f5-kube-api-access-5txw5" (OuterVolumeSpecName: "kube-api-access-5txw5") pod "ce2790a8-08d9-4dfb-baad-87abb63401f5" (UID: "ce2790a8-08d9-4dfb-baad-87abb63401f5"). InnerVolumeSpecName "kube-api-access-5txw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:39:10.812291 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.812262    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce2790a8-08d9-4dfb-baad-87abb63401f5-tls-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:10.812291 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.812287    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce2790a8-08d9-4dfb-baad-87abb63401f5-dshm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:10.812291 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.812296    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5txw5\" (UniqueName: \"kubernetes.io/projected/ce2790a8-08d9-4dfb-baad-87abb63401f5-kube-api-access-5txw5\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:10.860298 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.860265    2578 generic.go:356] "Generic (PLEG): container finished" podID="ce2790a8-08d9-4dfb-baad-87abb63401f5" containerID="953e9f89d69775adae06e606da32911b2c9702b26c55f7fc1094021e5d9213b6" exitCode=0
Jul 16 12:39:10.860444 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.860322    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"
Jul 16 12:39:10.860444 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.860330    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424" event={"ID":"ce2790a8-08d9-4dfb-baad-87abb63401f5","Type":"ContainerDied","Data":"953e9f89d69775adae06e606da32911b2c9702b26c55f7fc1094021e5d9213b6"}
Jul 16 12:39:10.860444 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.860363    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424" event={"ID":"ce2790a8-08d9-4dfb-baad-87abb63401f5","Type":"ContainerDied","Data":"3f79af2d08ebd077492e89a94379c36ed8b6752e45d1755fe0b7b6e743479dff"}
Jul 16 12:39:10.860444 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.860379    2578 scope.go:117] "RemoveContainer" containerID="953e9f89d69775adae06e606da32911b2c9702b26c55f7fc1094021e5d9213b6"
Jul 16 12:39:10.869243 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.869219    2578 scope.go:117] "RemoveContainer" containerID="953e9f89d69775adae06e606da32911b2c9702b26c55f7fc1094021e5d9213b6"
Jul 16 12:39:10.869487 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:39:10.869470    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"953e9f89d69775adae06e606da32911b2c9702b26c55f7fc1094021e5d9213b6\": container with ID starting with 953e9f89d69775adae06e606da32911b2c9702b26c55f7fc1094021e5d9213b6 not found: ID does not exist" containerID="953e9f89d69775adae06e606da32911b2c9702b26c55f7fc1094021e5d9213b6"
Jul 16 12:39:10.869575 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.869493    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"953e9f89d69775adae06e606da32911b2c9702b26c55f7fc1094021e5d9213b6"} err="failed to get container status \"953e9f89d69775adae06e606da32911b2c9702b26c55f7fc1094021e5d9213b6\": rpc error: code = NotFound desc = could not find container \"953e9f89d69775adae06e606da32911b2c9702b26c55f7fc1094021e5d9213b6\": container with ID starting with 953e9f89d69775adae06e606da32911b2c9702b26c55f7fc1094021e5d9213b6 not found: ID does not exist"
Jul 16 12:39:10.883348 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.883326    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"]
Jul 16 12:39:10.889024 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:10.889003    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a96583/rate-test-model-a96583-kserve-6dc6f5f64c-4s424"]
Jul 16 12:39:11.300238 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.300197    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce2790a8-08d9-4dfb-baad-87abb63401f5" path="/var/lib/kubelet/pods/ce2790a8-08d9-4dfb-baad-87abb63401f5/volumes"
Jul 16 12:39:11.606838 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.606714    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82" podUID="ee02e34a-6edc-490c-a984-8af21bd69bcf" containerName="main" containerID="cri-o://5cd586f2fcea9fbd5973e65f707e526f7d4563d66617f3ee6f7104bd0c091b4a" gracePeriod=45
Jul 16 12:39:11.867850 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.867785    2578 generic.go:356] "Generic (PLEG): container finished" podID="ee02e34a-6edc-490c-a984-8af21bd69bcf" containerID="5cd586f2fcea9fbd5973e65f707e526f7d4563d66617f3ee6f7104bd0c091b4a" exitCode=0
Jul 16 12:39:11.867978 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.867856    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82" event={"ID":"ee02e34a-6edc-490c-a984-8af21bd69bcf","Type":"ContainerDied","Data":"5cd586f2fcea9fbd5973e65f707e526f7d4563d66617f3ee6f7104bd0c091b4a"}
Jul 16 12:39:11.867978 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.867897    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82" event={"ID":"ee02e34a-6edc-490c-a984-8af21bd69bcf","Type":"ContainerDied","Data":"8ee8794f9f3d4f51f3069e163875611b525c43415009d2c1e7f8bef89826d23c"}
Jul 16 12:39:11.867978 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.867911    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ee8794f9f3d4f51f3069e163875611b525c43415009d2c1e7f8bef89826d23c"
Jul 16 12:39:11.868738 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.868718    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:39:11.921452 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.921421    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-tmp-dir\") pod \"ee02e34a-6edc-490c-a984-8af21bd69bcf\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") "
Jul 16 12:39:11.921452 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.921456    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njcjw\" (UniqueName: \"kubernetes.io/projected/ee02e34a-6edc-490c-a984-8af21bd69bcf-kube-api-access-njcjw\") pod \"ee02e34a-6edc-490c-a984-8af21bd69bcf\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") "
Jul 16 12:39:11.921692 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.921485    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-dshm\") pod \"ee02e34a-6edc-490c-a984-8af21bd69bcf\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") "
Jul 16 12:39:11.921692 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.921514    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee02e34a-6edc-490c-a984-8af21bd69bcf-tls-certs\") pod \"ee02e34a-6edc-490c-a984-8af21bd69bcf\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") "
Jul 16 12:39:11.921692 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.921554    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-home\") pod \"ee02e34a-6edc-490c-a984-8af21bd69bcf\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") "
Jul 16 12:39:11.921692 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.921604    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-model-cache\") pod \"ee02e34a-6edc-490c-a984-8af21bd69bcf\" (UID: \"ee02e34a-6edc-490c-a984-8af21bd69bcf\") "
Jul 16 12:39:11.921894 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.921732    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee02e34a-6edc-490c-a984-8af21bd69bcf" (UID: "ee02e34a-6edc-490c-a984-8af21bd69bcf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:39:11.921894 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.921842    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-home" (OuterVolumeSpecName: "home") pod "ee02e34a-6edc-490c-a984-8af21bd69bcf" (UID: "ee02e34a-6edc-490c-a984-8af21bd69bcf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:39:11.921894 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.921883    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee02e34a-6edc-490c-a984-8af21bd69bcf" (UID: "ee02e34a-6edc-490c-a984-8af21bd69bcf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:39:11.922006 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.921976    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-home\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:11.922006 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.921994    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:11.922098 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.922009    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-tmp-dir\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:11.923784 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.923758    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee02e34a-6edc-490c-a984-8af21bd69bcf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee02e34a-6edc-490c-a984-8af21bd69bcf" (UID: "ee02e34a-6edc-490c-a984-8af21bd69bcf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 12:39:11.923784 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.923760    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee02e34a-6edc-490c-a984-8af21bd69bcf-kube-api-access-njcjw" (OuterVolumeSpecName: "kube-api-access-njcjw") pod "ee02e34a-6edc-490c-a984-8af21bd69bcf" (UID: "ee02e34a-6edc-490c-a984-8af21bd69bcf"). InnerVolumeSpecName "kube-api-access-njcjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:39:11.923927 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:11.923767    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-dshm" (OuterVolumeSpecName: "dshm") pod "ee02e34a-6edc-490c-a984-8af21bd69bcf" (UID: "ee02e34a-6edc-490c-a984-8af21bd69bcf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:39:12.022768 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:12.022733    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njcjw\" (UniqueName: \"kubernetes.io/projected/ee02e34a-6edc-490c-a984-8af21bd69bcf-kube-api-access-njcjw\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:12.022768 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:12.022764    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee02e34a-6edc-490c-a984-8af21bd69bcf-dshm\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:12.022768 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:12.022773    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee02e34a-6edc-490c-a984-8af21bd69bcf-tls-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:12.874017 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:12.873987    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"
Jul 16 12:39:12.903757 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:12.903723    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"]
Jul 16 12:39:12.910661 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:12.910639    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-66e9a3/rate-test-model-66e9a3-kserve-7b66d875ff-5mk82"]
Jul 16 12:39:13.299855 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:13.299822    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee02e34a-6edc-490c-a984-8af21bd69bcf" path="/var/lib/kubelet/pods/ee02e34a-6edc-490c-a984-8af21bd69bcf/volumes"
Jul 16 12:39:41.708472 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:41.708439    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"]
Jul 16 12:39:41.709101 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:41.708641    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6" podUID="154fa342-6acf-4a63-832c-b78443bc7b17" containerName="istio-proxy" containerID="cri-o://6a304db283ae222a7e5af5a80e3c6ddc638fa3f58111e0464a8570bc8aafac0a" gracePeriod=30
Jul 16 12:39:46.977424 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:46.977366    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:39:47.051065 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.051013    2578 generic.go:356] "Generic (PLEG): container finished" podID="154fa342-6acf-4a63-832c-b78443bc7b17" containerID="6a304db283ae222a7e5af5a80e3c6ddc638fa3f58111e0464a8570bc8aafac0a" exitCode=0
Jul 16 12:39:47.051250 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.051070    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6" event={"ID":"154fa342-6acf-4a63-832c-b78443bc7b17","Type":"ContainerDied","Data":"6a304db283ae222a7e5af5a80e3c6ddc638fa3f58111e0464a8570bc8aafac0a"}
Jul 16 12:39:47.051250 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.051085    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"
Jul 16 12:39:47.051250 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.051122    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6" event={"ID":"154fa342-6acf-4a63-832c-b78443bc7b17","Type":"ContainerDied","Data":"c2a36607575a9a9efba170928deed6748a3f939fbe8f94bf5d4f42ba93ab2e05"}
Jul 16 12:39:47.051250 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.051146    2578 scope.go:117] "RemoveContainer" containerID="6a304db283ae222a7e5af5a80e3c6ddc638fa3f58111e0464a8570bc8aafac0a"
Jul 16 12:39:47.055402 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.055381    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-istio-envoy\") pod \"154fa342-6acf-4a63-832c-b78443bc7b17\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") "
Jul 16 12:39:47.055515 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.055409    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-credential-socket\") pod \"154fa342-6acf-4a63-832c-b78443bc7b17\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") "
Jul 16 12:39:47.055515 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.055463    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-istio-data\") pod \"154fa342-6acf-4a63-832c-b78443bc7b17\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") "
Jul 16 12:39:47.055515 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.055500    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz7vj\" (UniqueName: \"kubernetes.io/projected/154fa342-6acf-4a63-832c-b78443bc7b17-kube-api-access-dz7vj\") pod \"154fa342-6acf-4a63-832c-b78443bc7b17\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") "
Jul 16 12:39:47.055658 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.055554    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154fa342-6acf-4a63-832c-b78443bc7b17-istio-token\") pod \"154fa342-6acf-4a63-832c-b78443bc7b17\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") "
Jul 16 12:39:47.055658 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.055590    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-workload-socket\") pod \"154fa342-6acf-4a63-832c-b78443bc7b17\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") "
Jul 16 12:39:47.055658 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.055630    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-workload-certs\") pod \"154fa342-6acf-4a63-832c-b78443bc7b17\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") "
Jul 16 12:39:47.055816 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.055671    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154fa342-6acf-4a63-832c-b78443bc7b17-istio-podinfo\") pod \"154fa342-6acf-4a63-832c-b78443bc7b17\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") "
Jul 16 12:39:47.055816 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.055731    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154fa342-6acf-4a63-832c-b78443bc7b17-istiod-ca-cert\") pod \"154fa342-6acf-4a63-832c-b78443bc7b17\" (UID: \"154fa342-6acf-4a63-832c-b78443bc7b17\") "
Jul 16 12:39:47.055816 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.055738    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "154fa342-6acf-4a63-832c-b78443bc7b17" (UID: "154fa342-6acf-4a63-832c-b78443bc7b17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:39:47.056010 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.055986    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "154fa342-6acf-4a63-832c-b78443bc7b17" (UID: "154fa342-6acf-4a63-832c-b78443bc7b17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:39:47.056103 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.056001    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-credential-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:47.056273 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.056170    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "154fa342-6acf-4a63-832c-b78443bc7b17" (UID: "154fa342-6acf-4a63-832c-b78443bc7b17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:39:47.056473 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.056432    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/154fa342-6acf-4a63-832c-b78443bc7b17-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "154fa342-6acf-4a63-832c-b78443bc7b17" (UID: "154fa342-6acf-4a63-832c-b78443bc7b17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 12:39:47.057010 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.056979    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-istio-data" (OuterVolumeSpecName: "istio-data") pod "154fa342-6acf-4a63-832c-b78443bc7b17" (UID: "154fa342-6acf-4a63-832c-b78443bc7b17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:39:47.057864 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.057842    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "154fa342-6acf-4a63-832c-b78443bc7b17" (UID: "154fa342-6acf-4a63-832c-b78443bc7b17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:39:47.058242 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.058220    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154fa342-6acf-4a63-832c-b78443bc7b17-istio-token" (OuterVolumeSpecName: "istio-token") pod "154fa342-6acf-4a63-832c-b78443bc7b17" (UID: "154fa342-6acf-4a63-832c-b78443bc7b17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:39:47.058374 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.058351    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/154fa342-6acf-4a63-832c-b78443bc7b17-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "154fa342-6acf-4a63-832c-b78443bc7b17" (UID: "154fa342-6acf-4a63-832c-b78443bc7b17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 12:39:47.058438 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.058359    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154fa342-6acf-4a63-832c-b78443bc7b17-kube-api-access-dz7vj" (OuterVolumeSpecName: "kube-api-access-dz7vj") pod "154fa342-6acf-4a63-832c-b78443bc7b17" (UID: "154fa342-6acf-4a63-832c-b78443bc7b17"). InnerVolumeSpecName "kube-api-access-dz7vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:39:47.060341 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.060325    2578 scope.go:117] "RemoveContainer" containerID="b5215a1939469271acc5d344093b3152c722a7060cd4951ba92e1747498ea9ab"
Jul 16 12:39:47.076232 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.076212    2578 scope.go:117] "RemoveContainer" containerID="6a304db283ae222a7e5af5a80e3c6ddc638fa3f58111e0464a8570bc8aafac0a"
Jul 16 12:39:47.076489 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:39:47.076473    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a304db283ae222a7e5af5a80e3c6ddc638fa3f58111e0464a8570bc8aafac0a\": container with ID starting with 6a304db283ae222a7e5af5a80e3c6ddc638fa3f58111e0464a8570bc8aafac0a not found: ID does not exist" containerID="6a304db283ae222a7e5af5a80e3c6ddc638fa3f58111e0464a8570bc8aafac0a"
Jul 16 12:39:47.076535 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.076505    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a304db283ae222a7e5af5a80e3c6ddc638fa3f58111e0464a8570bc8aafac0a"} err="failed to get container status \"6a304db283ae222a7e5af5a80e3c6ddc638fa3f58111e0464a8570bc8aafac0a\": rpc error: code = NotFound desc = could not find container \"6a304db283ae222a7e5af5a80e3c6ddc638fa3f58111e0464a8570bc8aafac0a\": container with ID starting with 6a304db283ae222a7e5af5a80e3c6ddc638fa3f58111e0464a8570bc8aafac0a not found: ID does not exist"
Jul 16 12:39:47.076535 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.076524    2578 scope.go:117] "RemoveContainer" containerID="b5215a1939469271acc5d344093b3152c722a7060cd4951ba92e1747498ea9ab"
Jul 16 12:39:47.076777 ip-10-0-129-163 kubenswrapper[2578]: E0716 12:39:47.076757    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5215a1939469271acc5d344093b3152c722a7060cd4951ba92e1747498ea9ab\": container with ID starting with b5215a1939469271acc5d344093b3152c722a7060cd4951ba92e1747498ea9ab not found: ID does not exist" containerID="b5215a1939469271acc5d344093b3152c722a7060cd4951ba92e1747498ea9ab"
Jul 16 12:39:47.076823 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.076786    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5215a1939469271acc5d344093b3152c722a7060cd4951ba92e1747498ea9ab"} err="failed to get container status \"b5215a1939469271acc5d344093b3152c722a7060cd4951ba92e1747498ea9ab\": rpc error: code = NotFound desc = could not find container \"b5215a1939469271acc5d344093b3152c722a7060cd4951ba92e1747498ea9ab\": container with ID starting with b5215a1939469271acc5d344093b3152c722a7060cd4951ba92e1747498ea9ab not found: ID does not exist"
Jul 16 12:39:47.156800 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.156766    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-istio-data\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:47.156800 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.156794    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz7vj\" (UniqueName: \"kubernetes.io/projected/154fa342-6acf-4a63-832c-b78443bc7b17-kube-api-access-dz7vj\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:47.156800 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.156807    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154fa342-6acf-4a63-832c-b78443bc7b17-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:47.157033 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.156821    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-workload-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:47.157033 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.156832    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-workload-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:47.157033 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.156843    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154fa342-6acf-4a63-832c-b78443bc7b17-istio-podinfo\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:47.157033 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.156854    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154fa342-6acf-4a63-832c-b78443bc7b17-istiod-ca-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:47.157033 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.156865    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154fa342-6acf-4a63-832c-b78443bc7b17-istio-envoy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:39:47.372442 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.372412    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"]
Jul 16 12:39:47.377404 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:47.377379    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-a96583-openshift-default-5957f577b7-zk4k6"]
Jul 16 12:39:49.299494 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:39:49.299464    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="154fa342-6acf-4a63-832c-b78443bc7b17" path="/var/lib/kubelet/pods/154fa342-6acf-4a63-832c-b78443bc7b17/volumes"
Jul 16 12:40:08.019123 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:08.019088    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"]
Jul 16 12:40:08.019516 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:08.019336    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl" podUID="f45abf0a-af01-4622-8da8-ebd7d79c1bca" containerName="istio-proxy" containerID="cri-o://61a26169803ef65069f73a3ed453e65a92e01ad369f8f0c14d3e26db116400f2" gracePeriod=30
Jul 16 12:40:13.186286 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.186254    2578 generic.go:356] "Generic (PLEG): container finished" podID="f45abf0a-af01-4622-8da8-ebd7d79c1bca" containerID="61a26169803ef65069f73a3ed453e65a92e01ad369f8f0c14d3e26db116400f2" exitCode=0
Jul 16 12:40:13.186646 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.186319    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl" event={"ID":"f45abf0a-af01-4622-8da8-ebd7d79c1bca","Type":"ContainerDied","Data":"61a26169803ef65069f73a3ed453e65a92e01ad369f8f0c14d3e26db116400f2"}
Jul 16 12:40:13.186646 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.186376    2578 scope.go:117] "RemoveContainer" containerID="e2e7741462a31973a645893587a154f636a20bedf6374728a36bb98a8be2d259"
Jul 16 12:40:13.284055 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.283984    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:40:13.395755 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.395724    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istiod-ca-cert\") pod \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") "
Jul 16 12:40:13.395755 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.395759    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-data\") pod \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") "
Jul 16 12:40:13.396057 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.395798    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-workload-certs\") pod \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") "
Jul 16 12:40:13.396057 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.395839    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-workload-socket\") pod \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") "
Jul 16 12:40:13.396057 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.395861    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-token\") pod \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") "
Jul 16 12:40:13.396057 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.395881    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-credential-socket\") pod \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") "
Jul 16 12:40:13.396057 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.395903    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-envoy\") pod \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") "
Jul 16 12:40:13.396057 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.395924    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mclw2\" (UniqueName: \"kubernetes.io/projected/f45abf0a-af01-4622-8da8-ebd7d79c1bca-kube-api-access-mclw2\") pod \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") "
Jul 16 12:40:13.396057 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.395989    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-podinfo\") pod \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\" (UID: \"f45abf0a-af01-4622-8da8-ebd7d79c1bca\") "
Jul 16 12:40:13.396426 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.396143    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f45abf0a-af01-4622-8da8-ebd7d79c1bca" (UID: "f45abf0a-af01-4622-8da8-ebd7d79c1bca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 12:40:13.396426 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.396202    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f45abf0a-af01-4622-8da8-ebd7d79c1bca" (UID: "f45abf0a-af01-4622-8da8-ebd7d79c1bca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:40:13.396426 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.396397    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istiod-ca-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:40:13.396426 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.396419    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-workload-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:40:13.396621 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.396398    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f45abf0a-af01-4622-8da8-ebd7d79c1bca" (UID: "f45abf0a-af01-4622-8da8-ebd7d79c1bca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:40:13.396621 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.396480    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f45abf0a-af01-4622-8da8-ebd7d79c1bca" (UID: "f45abf0a-af01-4622-8da8-ebd7d79c1bca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:40:13.396770 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.396748    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-data" (OuterVolumeSpecName: "istio-data") pod "f45abf0a-af01-4622-8da8-ebd7d79c1bca" (UID: "f45abf0a-af01-4622-8da8-ebd7d79c1bca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:40:13.398353 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.398328    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45abf0a-af01-4622-8da8-ebd7d79c1bca-kube-api-access-mclw2" (OuterVolumeSpecName: "kube-api-access-mclw2") pod "f45abf0a-af01-4622-8da8-ebd7d79c1bca" (UID: "f45abf0a-af01-4622-8da8-ebd7d79c1bca"). InnerVolumeSpecName "kube-api-access-mclw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:40:13.398460 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.398366    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f45abf0a-af01-4622-8da8-ebd7d79c1bca" (UID: "f45abf0a-af01-4622-8da8-ebd7d79c1bca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 12:40:13.398636 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.398606    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-token" (OuterVolumeSpecName: "istio-token") pod "f45abf0a-af01-4622-8da8-ebd7d79c1bca" (UID: "f45abf0a-af01-4622-8da8-ebd7d79c1bca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 12:40:13.398636 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.398615    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f45abf0a-af01-4622-8da8-ebd7d79c1bca" (UID: "f45abf0a-af01-4622-8da8-ebd7d79c1bca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 12:40:13.497594 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.497567    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-workload-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:40:13.497594 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.497590    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-token\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:40:13.497594 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.497600    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-credential-socket\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:40:13.497809 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.497609    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-envoy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:40:13.497809 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.497617    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mclw2\" (UniqueName: \"kubernetes.io/projected/f45abf0a-af01-4622-8da8-ebd7d79c1bca-kube-api-access-mclw2\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:40:13.497809 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.497626    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-podinfo\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:40:13.497809 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:13.497634    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45abf0a-af01-4622-8da8-ebd7d79c1bca-istio-data\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 16 12:40:14.193344 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:14.193308    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl" event={"ID":"f45abf0a-af01-4622-8da8-ebd7d79c1bca","Type":"ContainerDied","Data":"0d2c6d7554cc11365426d1c0e0537a38d55a5f8e7a328fe2ef0dc59369e9de7c"}
Jul 16 12:40:14.193811 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:14.193376    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"
Jul 16 12:40:14.193811 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:14.193382    2578 scope.go:117] "RemoveContainer" containerID="61a26169803ef65069f73a3ed453e65a92e01ad369f8f0c14d3e26db116400f2"
Jul 16 12:40:14.215756 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:14.215730    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"]
Jul 16 12:40:14.222733 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:14.222710    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-66e9a3-openshift-default-58b975d5c9-2m5hl"]
Jul 16 12:40:14.809189 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:14.809162    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-qqwbh_4973c159-ec99-42a4-9114-388ef2c6036a/manager/0.log"
Jul 16 12:40:14.941982 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:14.941941    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-c99d7_78107917-5ba9-4339-bd60-d0ee4ab408b4/manager/0.log"
Jul 16 12:40:15.299673 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:15.299636    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f45abf0a-af01-4622-8da8-ebd7d79c1bca" path="/var/lib/kubelet/pods/f45abf0a-af01-4622-8da8-ebd7d79c1bca/volumes"
Jul 16 12:40:15.319108 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:15.319076    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-95sxr_36092f20-8fe5-4513-aa16-b10fcaaeed44/manager/0.log"
Jul 16 12:40:15.446713 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:15.446688    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57d7447f8f-75zrk_36a058c0-7c4b-4724-badc-788bb9c48bd0/copy-manifests/0.log"
Jul 16 12:40:15.453664 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:15.453637    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57d7447f8f-75zrk_36a058c0-7c4b-4724-badc-788bb9c48bd0/manager/0.log"
Jul 16 12:40:19.479092 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:19.479064    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-9dczj_cbaaae31-9490-4c9d-8ad7-21e84cf8c0db/manager/0.log"
Jul 16 12:40:19.606484 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:19.606458    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-n9mmp_90410e5d-cd0c-4db8-9e2a-5f2cbe83d529/manager/0.log"
Jul 16 12:40:19.730820 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:19.730741    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-s7rhm_73e603da-fca7-4e9f-82a2-25b0c7b3076c/kuadrant-console-plugin/0.log"
Jul 16 12:40:19.863207 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:19.863172    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-gb4pb_3c23f979-c73c-4424-916e-cf0229186561/registry-server/0.log"
Jul 16 12:40:20.000525 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:20.000452    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-gtqx2_d30358f9-a62b-4fb1-8df4-374c0419d33a/manager/0.log"
Jul 16 12:40:20.257822 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:20.257745    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-72mxq_ab96b12a-1d9b-41fa-b16c-4193c471338d/manager/0.log"
Jul 16 12:40:20.638244 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:20.638197    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6785nq4x_52d7bf4e-b9d9-4b85-a599-8976e5cddd3a/istio-proxy/0.log"
Jul 16 12:40:20.890362 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:20.890279    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69fc9c4ddf-6487s_2372c486-6a35-4746-9419-b75cb62ec5cd/kube-auth-proxy/0.log"
Jul 16 12:40:21.144543 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:21.144448    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-bzksk_2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421/istio-proxy/1.log"
Jul 16 12:40:21.533592 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:21.533515    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85b9cb9c46-gmf4x_f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e/router/0.log"
Jul 16 12:40:22.153595 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:22.153557    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-pc6bk_e3906585-214d-4a81-914c-3f2807b41cfa/main/0.log"
Jul 16 12:40:22.276878 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:22.276824    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6ck4t_da4d20e4-6edf-43e3-ae6c-ccc3aaa5b820/main/0.log"
Jul 16 12:40:22.407785 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:22.407710    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-n27gv_0866ac5e-6a94-4de0-85ec-e120a84159cc/main/0.log"
Jul 16 12:40:31.678130 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:31.678101    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w5wfm_32fffc0e-1b8f-44f1-ab44-b4b0ac9820d7/global-pull-secret-syncer/0.log"
Jul 16 12:40:31.759829 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:31.759798    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mv5xr_dc1f2e2f-da1b-43e6-abd5-87ec82aee7e3/konnectivity-agent/0.log"
Jul 16 12:40:31.829620 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:31.829591    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-163.ec2.internal_84c1f669f09517dfcf8e52b12af2bb23/haproxy/0.log"
Jul 16 12:40:37.566825 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:37.566787    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-9dczj_cbaaae31-9490-4c9d-8ad7-21e84cf8c0db/manager/0.log"
Jul 16 12:40:37.599249 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:37.599221    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-n9mmp_90410e5d-cd0c-4db8-9e2a-5f2cbe83d529/manager/0.log"
Jul 16 12:40:37.638699 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:37.638672    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-s7rhm_73e603da-fca7-4e9f-82a2-25b0c7b3076c/kuadrant-console-plugin/0.log"
Jul 16 12:40:37.680773 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:37.680741    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-gb4pb_3c23f979-c73c-4424-916e-cf0229186561/registry-server/0.log"
Jul 16 12:40:37.802449 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:37.802418    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-gtqx2_d30358f9-a62b-4fb1-8df4-374c0419d33a/manager/0.log"
Jul 16 12:40:37.925641 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:37.925563    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-72mxq_ab96b12a-1d9b-41fa-b16c-4193c471338d/manager/0.log"
Jul 16 12:40:39.843422 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:39.843389    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-vpthq_715b8182-f04a-4e70-9766-1f4075a48dab/cluster-monitoring-operator/0.log"
Jul 16 12:40:40.190584 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:40.190499    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zkrrr_a3bd6541-f3e1-41ac-8693-ba50e1366d9b/node-exporter/0.log"
Jul 16 12:40:40.213459 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:40.213433    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zkrrr_a3bd6541-f3e1-41ac-8693-ba50e1366d9b/kube-rbac-proxy/0.log"
Jul 16 12:40:40.241292 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:40.241261    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zkrrr_a3bd6541-f3e1-41ac-8693-ba50e1366d9b/init-textfile/0.log"
Jul 16 12:40:42.638049 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.637998    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"]
Jul 16 12:40:42.638434 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.638362    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="154fa342-6acf-4a63-832c-b78443bc7b17" containerName="istio-proxy"
Jul 16 12:40:42.638434 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.638375    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="154fa342-6acf-4a63-832c-b78443bc7b17" containerName="istio-proxy"
Jul 16 12:40:42.638434 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.638383    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee02e34a-6edc-490c-a984-8af21bd69bcf" containerName="main"
Jul 16 12:40:42.638434 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.638391    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee02e34a-6edc-490c-a984-8af21bd69bcf" containerName="main"
Jul 16 12:40:42.638434 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.638402    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f45abf0a-af01-4622-8da8-ebd7d79c1bca" containerName="istio-proxy"
Jul 16 12:40:42.638434 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.638408    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f45abf0a-af01-4622-8da8-ebd7d79c1bca" containerName="istio-proxy"
Jul 16 12:40:42.638434 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.638421    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="154fa342-6acf-4a63-832c-b78443bc7b17" containerName="istio-proxy"
Jul 16 12:40:42.638434 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.638427    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="154fa342-6acf-4a63-832c-b78443bc7b17" containerName="istio-proxy"
Jul 16 12:40:42.638434 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.638436    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce2790a8-08d9-4dfb-baad-87abb63401f5" containerName="main"
Jul 16 12:40:42.638713 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.638441    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce2790a8-08d9-4dfb-baad-87abb63401f5" containerName="main"
Jul 16 12:40:42.638713 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.638510    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce2790a8-08d9-4dfb-baad-87abb63401f5" containerName="main"
Jul 16 12:40:42.638713 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.638518    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f45abf0a-af01-4622-8da8-ebd7d79c1bca" containerName="istio-proxy"
Jul 16 12:40:42.638713 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.638523    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee02e34a-6edc-490c-a984-8af21bd69bcf" containerName="main"
Jul 16 12:40:42.638713 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.638530    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="154fa342-6acf-4a63-832c-b78443bc7b17" containerName="istio-proxy"
Jul 16 12:40:42.638713 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.638536    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="154fa342-6acf-4a63-832c-b78443bc7b17" containerName="istio-proxy"
Jul 16 12:40:42.641368 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.641348    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"
Jul 16 12:40:42.644187 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.644161    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2sc8x\"/\"kube-root-ca.crt\""
Jul 16 12:40:42.645029 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.645009    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2sc8x\"/\"default-dockercfg-hxnm7\""
Jul 16 12:40:42.645155 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.645065    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2sc8x\"/\"openshift-service-ca.crt\""
Jul 16 12:40:42.650821 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.650799    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"]
Jul 16 12:40:42.755840 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.755804    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bc9efff-5adb-4eca-9c86-9ac49de1ec41-sys\") pod \"perf-node-gather-daemonset-999gh\" (UID: \"8bc9efff-5adb-4eca-9c86-9ac49de1ec41\") " pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"
Jul 16 12:40:42.755840 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.755843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8bc9efff-5adb-4eca-9c86-9ac49de1ec41-proc\") pod \"perf-node-gather-daemonset-999gh\" (UID: \"8bc9efff-5adb-4eca-9c86-9ac49de1ec41\") " pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"
Jul 16 12:40:42.756127 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.755954    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bc9efff-5adb-4eca-9c86-9ac49de1ec41-lib-modules\") pod \"perf-node-gather-daemonset-999gh\" (UID: \"8bc9efff-5adb-4eca-9c86-9ac49de1ec41\") " pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"
Jul 16 12:40:42.756127 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.755987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8bc9efff-5adb-4eca-9c86-9ac49de1ec41-podres\") pod \"perf-node-gather-daemonset-999gh\" (UID: \"8bc9efff-5adb-4eca-9c86-9ac49de1ec41\") " pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"
Jul 16 12:40:42.756127 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.756013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x4q2\" (UniqueName: \"kubernetes.io/projected/8bc9efff-5adb-4eca-9c86-9ac49de1ec41-kube-api-access-9x4q2\") pod \"perf-node-gather-daemonset-999gh\" (UID: \"8bc9efff-5adb-4eca-9c86-9ac49de1ec41\") " pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"
Jul 16 12:40:42.856939 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.856885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bc9efff-5adb-4eca-9c86-9ac49de1ec41-lib-modules\") pod \"perf-node-gather-daemonset-999gh\" (UID: \"8bc9efff-5adb-4eca-9c86-9ac49de1ec41\") " pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"
Jul 16 12:40:42.856939 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.856942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8bc9efff-5adb-4eca-9c86-9ac49de1ec41-podres\") pod \"perf-node-gather-daemonset-999gh\" (UID: \"8bc9efff-5adb-4eca-9c86-9ac49de1ec41\") " pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"
Jul 16 12:40:42.857225 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.856964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x4q2\" (UniqueName: \"kubernetes.io/projected/8bc9efff-5adb-4eca-9c86-9ac49de1ec41-kube-api-access-9x4q2\") pod \"perf-node-gather-daemonset-999gh\" (UID: \"8bc9efff-5adb-4eca-9c86-9ac49de1ec41\") " pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"
Jul 16 12:40:42.857225 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.857000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bc9efff-5adb-4eca-9c86-9ac49de1ec41-sys\") pod \"perf-node-gather-daemonset-999gh\" (UID: \"8bc9efff-5adb-4eca-9c86-9ac49de1ec41\") " pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"
Jul 16 12:40:42.857225 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.857015    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8bc9efff-5adb-4eca-9c86-9ac49de1ec41-proc\") pod \"perf-node-gather-daemonset-999gh\" (UID: \"8bc9efff-5adb-4eca-9c86-9ac49de1ec41\") " pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"
Jul 16 12:40:42.857225 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.857107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8bc9efff-5adb-4eca-9c86-9ac49de1ec41-proc\") pod \"perf-node-gather-daemonset-999gh\" (UID: \"8bc9efff-5adb-4eca-9c86-9ac49de1ec41\") " pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"
Jul 16 12:40:42.857225 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.857110    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bc9efff-5adb-4eca-9c86-9ac49de1ec41-lib-modules\") pod \"perf-node-gather-daemonset-999gh\" (UID: \"8bc9efff-5adb-4eca-9c86-9ac49de1ec41\") " pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"
Jul 16 12:40:42.857225 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.857149    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8bc9efff-5adb-4eca-9c86-9ac49de1ec41-podres\") pod \"perf-node-gather-daemonset-999gh\" (UID: \"8bc9efff-5adb-4eca-9c86-9ac49de1ec41\") " pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"
Jul 16 12:40:42.857225 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.857168    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bc9efff-5adb-4eca-9c86-9ac49de1ec41-sys\") pod \"perf-node-gather-daemonset-999gh\" (UID: \"8bc9efff-5adb-4eca-9c86-9ac49de1ec41\") " pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"
Jul 16 12:40:42.866946 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.866921    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x4q2\" (UniqueName: \"kubernetes.io/projected/8bc9efff-5adb-4eca-9c86-9ac49de1ec41-kube-api-access-9x4q2\") pod \"perf-node-gather-daemonset-999gh\" (UID: \"8bc9efff-5adb-4eca-9c86-9ac49de1ec41\") " pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"
Jul 16 12:40:42.951940 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:42.951850    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"
Jul 16 12:40:43.084317 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:43.084279    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"]
Jul 16 12:40:43.087581 ip-10-0-129-163 kubenswrapper[2578]: W0716 12:40:43.087550    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8bc9efff_5adb_4eca_9c86_9ac49de1ec41.slice/crio-2a965817133e1197286c74ebecea26ec6afde2e78a61f64025acca0978aa9103 WatchSource:0}: Error finding container 2a965817133e1197286c74ebecea26ec6afde2e78a61f64025acca0978aa9103: Status 404 returned error can't find the container with id 2a965817133e1197286c74ebecea26ec6afde2e78a61f64025acca0978aa9103
Jul 16 12:40:43.157417 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:43.157396    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-bqhbq_940b7de4-6a55-4b35-8c54-adb00b18b803/download-server/0.log"
Jul 16 12:40:43.331966 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:43.331929    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh" event={"ID":"8bc9efff-5adb-4eca-9c86-9ac49de1ec41","Type":"ContainerStarted","Data":"d2b57adbf3700bd853c772856d1e7c73bb6da3f1c960d63c40c9f49b4d1e9525"}
Jul 16 12:40:43.332175 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:43.331983    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh" event={"ID":"8bc9efff-5adb-4eca-9c86-9ac49de1ec41","Type":"ContainerStarted","Data":"2a965817133e1197286c74ebecea26ec6afde2e78a61f64025acca0978aa9103"}
Jul 16 12:40:43.332175 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:43.332020    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"
Jul 16 12:40:43.350404 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:43.350356    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh" podStartSLOduration=1.350342585 podStartE2EDuration="1.350342585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:40:43.348916165 +0000 UTC m=+2984.649693289" watchObservedRunningTime="2026-07-16 12:40:43.350342585 +0000 UTC m=+2984.651119682"
Jul 16 12:40:43.728017 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:43.727933    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-z5cnp_ce18ca5b-8314-4db8-9fc6-7717fcb0b0cc/volume-data-source-validator/0.log"
Jul 16 12:40:44.912980 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:44.912951    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qm4wx_2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a/dns/0.log"
Jul 16 12:40:44.936242 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:44.936219    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qm4wx_2e8ed3c2-9f66-4cb3-be0d-a58c0f1f1a5a/kube-rbac-proxy/0.log"
Jul 16 12:40:44.963015 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:44.962992    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4sj9h_8b123f16-31a0-49be-906c-017bdc2602e6/dns-node-resolver/0.log"
Jul 16 12:40:45.657364 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:45.657335    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vlbks_c5d99cb5-959d-4a75-89e7-fa68635e416d/node-ca/0.log"
Jul 16 12:40:46.622513 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:46.622484    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6785nq4x_52d7bf4e-b9d9-4b85-a599-8976e5cddd3a/istio-proxy/0.log"
Jul 16 12:40:46.766732 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:46.766702    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69fc9c4ddf-6487s_2372c486-6a35-4746-9419-b75cb62ec5cd/kube-auth-proxy/0.log"
Jul 16 12:40:46.824582 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:46.824550    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-bzksk_2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421/istio-proxy/0.log"
Jul 16 12:40:46.851193 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:46.851152    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-bzksk_2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421/istio-proxy/1.log"
Jul 16 12:40:46.934991 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:46.934909    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85b9cb9c46-gmf4x_f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e/router/0.log"
Jul 16 12:40:47.693503 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:47.693473    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-crzvg_af3a84f5-1b60-40db-acd1-3624b84c09fa/serve-healthcheck-canary/0.log"
Jul 16 12:40:48.206547 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:48.206522    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-qmc55_64f7c050-5fde-47f1-9e42-5deaca93acd7/insights-operator/0.log"
Jul 16 12:40:48.208844 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:48.208828    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-qmc55_64f7c050-5fde-47f1-9e42-5deaca93acd7/insights-operator/1.log"
Jul 16 12:40:48.383239 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:48.383199    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rr62s_81f767f1-2e6f-4ad1-b92d-612c527a97b3/kube-rbac-proxy/0.log"
Jul 16 12:40:48.410155 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:48.410123    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rr62s_81f767f1-2e6f-4ad1-b92d-612c527a97b3/exporter/0.log"
Jul 16 12:40:48.433445 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:48.433415    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rr62s_81f767f1-2e6f-4ad1-b92d-612c527a97b3/extractor/0.log"
Jul 16 12:40:49.346998 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:49.346970    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2sc8x/perf-node-gather-daemonset-999gh"
Jul 16 12:40:50.623632 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:50.623605    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-qqwbh_4973c159-ec99-42a4-9114-388ef2c6036a/manager/0.log"
Jul 16 12:40:50.762614 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:50.762582    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-c99d7_78107917-5ba9-4339-bd60-d0ee4ab408b4/manager/0.log"
Jul 16 12:40:50.957677 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:50.957601    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-95sxr_36092f20-8fe5-4513-aa16-b10fcaaeed44/manager/0.log"
Jul 16 12:40:50.985741 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:50.985716    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57d7447f8f-75zrk_36a058c0-7c4b-4724-badc-788bb9c48bd0/manager/0.log"
Jul 16 12:40:51.012399 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:51.012367    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57d7447f8f-75zrk_36a058c0-7c4b-4724-badc-788bb9c48bd0/copy-manifests/0.log"
Jul 16 12:40:52.351686 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:52.351657    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6785nq4x_52d7bf4e-b9d9-4b85-a599-8976e5cddd3a/istio-proxy/0.log"
Jul 16 12:40:52.496737 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:52.496708    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69fc9c4ddf-6487s_2372c486-6a35-4746-9419-b75cb62ec5cd/kube-auth-proxy/0.log"
Jul 16 12:40:52.571179 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:52.571146    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-bzksk_2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421/istio-proxy/0.log"
Jul 16 12:40:52.601615 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:52.601586    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-bzksk_2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421/istio-proxy/1.log"
Jul 16 12:40:52.671318 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:52.671230    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85b9cb9c46-gmf4x_f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e/router/0.log"
Jul 16 12:40:53.987925 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:53.987883    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6785nq4x_52d7bf4e-b9d9-4b85-a599-8976e5cddd3a/istio-proxy/0.log"
Jul 16 12:40:54.116141 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:54.116061    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69fc9c4ddf-6487s_2372c486-6a35-4746-9419-b75cb62ec5cd/kube-auth-proxy/0.log"
Jul 16 12:40:54.158746 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:54.158716    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-bzksk_2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421/istio-proxy/0.log"
Jul 16 12:40:54.196640 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:54.196613    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-bzksk_2cf9ca58-d9ac-4b47-9aa3-2a177fbb0421/istio-proxy/1.log"
Jul 16 12:40:54.258163 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:54.258138    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85b9cb9c46-gmf4x_f01a2273-37fb-49cb-a9a3-17ebbd6f8f3e/router/0.log"
Jul 16 12:40:59.308582 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:59.308555    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgflz_4c9ae01e-f7f2-4296-a0b0-600bd31f36a8/kube-multus-additional-cni-plugins/0.log"
Jul 16 12:40:59.329527 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:59.329502    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgflz_4c9ae01e-f7f2-4296-a0b0-600bd31f36a8/egress-router-binary-copy/0.log"
Jul 16 12:40:59.351958 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:59.351940    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgflz_4c9ae01e-f7f2-4296-a0b0-600bd31f36a8/cni-plugins/0.log"
Jul 16 12:40:59.374552 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:59.374531    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgflz_4c9ae01e-f7f2-4296-a0b0-600bd31f36a8/bond-cni-plugin/0.log"
Jul 16 12:40:59.398943 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:59.398926    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgflz_4c9ae01e-f7f2-4296-a0b0-600bd31f36a8/routeoverride-cni/0.log"
Jul 16 12:40:59.421867 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:59.421849    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgflz_4c9ae01e-f7f2-4296-a0b0-600bd31f36a8/whereabouts-cni-bincopy/0.log"
Jul 16 12:40:59.449252 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:59.449232    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgflz_4c9ae01e-f7f2-4296-a0b0-600bd31f36a8/whereabouts-cni/0.log"
Jul 16 12:40:59.732199 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:59.732104    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-z7mwh_7dfeaf25-6b49-467f-ae5d-5e0ea9c503e6/kube-multus/0.log"
Jul 16 12:40:59.755007 ip-10-0-129-163 kubenswrapper[2578]: I0716 12:40:59.754982    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4p7gl_18353093-2e17-408e-b541-5dfb339343f0/network-metrics-daemon/0.log"