Jul 15 12:11:16.262051 ip-10-0-141-45 systemd[1]: Starting Kubernetes Kubelet... Jul 15 12:11:16.611220 ip-10-0-141-45 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 12:11:16.611220 ip-10-0-141-45 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 12:11:16.611220 ip-10-0-141-45 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 12:11:16.611220 ip-10-0-141-45 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 12:11:16.611220 ip-10-0-141-45 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 12:11:16.612861 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.612766 2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 12:11:16.617086 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617070 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 12:11:16.617086 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617086 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 12:11:16.617153 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617090 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 12:11:16.617153 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617094 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 12:11:16.617153 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617096 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 15 12:11:16.617153 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617099 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 12:11:16.617153 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617102 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 12:11:16.617153 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617105 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 12:11:16.617153 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617108 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 12:11:16.617153 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617111 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 12:11:16.617153 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617114 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 12:11:16.617153 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617117 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 12:11:16.617153 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617119 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 12:11:16.617153 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617124 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 12:11:16.617153 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617128 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 12:11:16.617153 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617131 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 12:11:16.617153 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617133 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 12:11:16.617153 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617136 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 12:11:16.617153 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617139 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 12:11:16.617153 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617149 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 12:11:16.617153 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617152 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 12:11:16.617589 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617155 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 12:11:16.617589 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617157 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 12:11:16.617589 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617160 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 12:11:16.617589 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617163 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 12:11:16.617589 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617166 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 12:11:16.617589 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617169 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 12:11:16.617589 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617171 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 12:11:16.617589 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617174 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 12:11:16.617589 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617176 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 12:11:16.617589 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617179 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 12:11:16.617589 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617181 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 12:11:16.617589 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617183 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 12:11:16.617589 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617186 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 12:11:16.617589 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617189 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 12:11:16.617589 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617191 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 12:11:16.617589 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617194 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 12:11:16.617589 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617196 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 12:11:16.617589 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617199 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 12:11:16.617589 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617202 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 12:11:16.617589 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617204 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 12:11:16.618089 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617206 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 12:11:16.618089 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617208 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 12:11:16.618089 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617211 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 12:11:16.618089 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617213 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 12:11:16.618089 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617215 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 12:11:16.618089 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617218 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 12:11:16.618089 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617222 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 12:11:16.618089 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617226 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 12:11:16.618089 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617229 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 12:11:16.618089 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617231 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 12:11:16.618089 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617235 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 12:11:16.618089 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617238 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 12:11:16.618089 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617241 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 12:11:16.618089 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617244 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 12:11:16.618089 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617248 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 12:11:16.618089 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617250 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 12:11:16.618089 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617253 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 12:11:16.618089 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617255 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 12:11:16.618089 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617257 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 12:11:16.618089 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617260 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 12:11:16.618603 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617262 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 12:11:16.618603 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617265 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 12:11:16.618603 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617269 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 12:11:16.618603 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617273 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 12:11:16.618603 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617275 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 12:11:16.618603 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617278 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 12:11:16.618603 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617280 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 12:11:16.618603 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617283 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 12:11:16.618603 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617285 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 12:11:16.618603 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617288 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 12:11:16.618603 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617290 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 12:11:16.618603 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617292 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 12:11:16.618603 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617295 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 12:11:16.618603 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617297 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 12:11:16.618603 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617300 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 12:11:16.618603 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617302 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 12:11:16.618603 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617305 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 12:11:16.618603 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617307 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 12:11:16.618603 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617309 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 12:11:16.619065 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617312 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 12:11:16.619065 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617315 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 12:11:16.619065 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617317 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 12:11:16.619065 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617319 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 12:11:16.619065 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617323 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 12:11:16.619065 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617326 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 12:11:16.619065 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617328 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 12:11:16.619065 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617331 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 12:11:16.619065 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617334 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 12:11:16.619065 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617336 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 12:11:16.619065 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617339 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 12:11:16.619065 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617341 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 12:11:16.619065 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617344 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 12:11:16.619065 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617346 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 12:11:16.619065 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617348 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 12:11:16.619065 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617351 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 12:11:16.619065 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617354 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 12:11:16.619065 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617356 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 12:11:16.619065 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.617358 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 12:11:16.619065 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618050 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 12:11:16.619542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618056 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 12:11:16.619542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618059 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 12:11:16.619542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618062 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 12:11:16.619542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618065 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 12:11:16.619542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618067 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 12:11:16.619542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618070 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 12:11:16.619542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618072 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 12:11:16.619542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618075 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 12:11:16.619542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618078 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 12:11:16.619542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618080 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 12:11:16.619542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618083 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 12:11:16.619542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618085 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 12:11:16.619542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618088 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 12:11:16.619542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618091 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 12:11:16.619542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618093 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 12:11:16.619542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618096 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 12:11:16.619542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618099 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 12:11:16.619542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618103 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 12:11:16.619542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618106 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 12:11:16.619542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618108 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 12:11:16.620043 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618111 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 12:11:16.620043 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618114 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 12:11:16.620043 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618116 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 12:11:16.620043 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618119 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 12:11:16.620043 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618122 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 12:11:16.620043 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618124 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 12:11:16.620043 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618127 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 12:11:16.620043 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618129 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 12:11:16.620043 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618132 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 12:11:16.620043 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618134 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 12:11:16.620043 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618136 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 12:11:16.620043 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618139 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 12:11:16.620043 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618141 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 12:11:16.620043 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618144 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 15 12:11:16.620043 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618146 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 12:11:16.620043 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618148 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 12:11:16.620043 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618151 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 12:11:16.620043 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618154 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 12:11:16.620043 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618159 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 12:11:16.620490 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618164 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 12:11:16.620490 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618167 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 12:11:16.620490 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618170 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 12:11:16.620490 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618174 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 12:11:16.620490 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618177 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 12:11:16.620490 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618179 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 12:11:16.620490 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618182 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 12:11:16.620490 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618185 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 12:11:16.620490 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618187 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 12:11:16.620490 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618191 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 12:11:16.620490 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618194 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 12:11:16.620490 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618200 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 12:11:16.620490 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618203 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 12:11:16.620490 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618206 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 12:11:16.620490 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618209 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 12:11:16.620490 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618212 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 12:11:16.620490 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618215 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 12:11:16.620490 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618219 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 12:11:16.620490 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618223 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 12:11:16.620963 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618226 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 12:11:16.620963 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618229 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 12:11:16.620963 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618232 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 12:11:16.620963 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618234 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 12:11:16.620963 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618237 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 12:11:16.620963 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618240 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 12:11:16.620963 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618243 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 12:11:16.620963 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618246 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 12:11:16.620963 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618248 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 12:11:16.620963 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618251 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 12:11:16.620963 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618253 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 12:11:16.620963 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618255 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 12:11:16.620963 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618258 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 12:11:16.620963 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618261 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 12:11:16.620963 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618263 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 12:11:16.620963 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618265 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 12:11:16.620963 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618267 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 12:11:16.620963 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618270 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 12:11:16.620963 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618272 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 12:11:16.620963 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618274 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 12:11:16.621440 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618277 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 12:11:16.621440 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618279 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 12:11:16.621440 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618282 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 12:11:16.621440 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618285 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 12:11:16.621440 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618288 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 12:11:16.621440 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618291 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 12:11:16.621440 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618293 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 12:11:16.621440 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618296 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 12:11:16.621440 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618298 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 12:11:16.621440 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618301 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 12:11:16.621440 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618303 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 12:11:16.621440 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618306 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 12:11:16.621440 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618308 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 12:11:16.621440 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618311 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 12:11:16.621440 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618313 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 12:11:16.621440 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618315 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 12:11:16.621440 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618318 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 12:11:16.621440 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618320 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 12:11:16.621440 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618322 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 12:11:16.621440 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618325 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 12:11:16.621952 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618694 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 12:11:16.621952 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618698 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 12:11:16.621952 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618701 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 12:11:16.621952 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618704 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 12:11:16.621952 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618706 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 12:11:16.621952 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618709 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 12:11:16.621952 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618711 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 12:11:16.621952 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618714 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 12:11:16.621952 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618717 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 12:11:16.621952 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618719 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 12:11:16.621952 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618722 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 12:11:16.621952 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618724 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 12:11:16.621952 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618726 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 12:11:16.621952 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618729 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 12:11:16.621952 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618733 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 12:11:16.621952 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618737 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 12:11:16.621952 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618740 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 12:11:16.621952 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618743 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 12:11:16.621952 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618747 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 12:11:16.622407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618749 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 12:11:16.622407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618752 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 12:11:16.622407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618755 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 12:11:16.622407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618758 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 12:11:16.622407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618760 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 12:11:16.622407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618763 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 12:11:16.622407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618766 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 12:11:16.622407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618785 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 12:11:16.622407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618789 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 15 12:11:16.622407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618791 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 12:11:16.622407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618794 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 12:11:16.622407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618797 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 12:11:16.622407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618799 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 12:11:16.622407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618802 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 12:11:16.622407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618804 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 12:11:16.622407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618807 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 12:11:16.622407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618810 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 12:11:16.622407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618812 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 12:11:16.622407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618815 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 12:11:16.622407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618818 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 12:11:16.622904 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618821 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 12:11:16.622904 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618824 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 12:11:16.622904 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618826 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 12:11:16.622904 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618829 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 12:11:16.622904 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618831 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 12:11:16.622904 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618834 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 12:11:16.622904 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618836 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 12:11:16.622904 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618839 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 12:11:16.622904 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618842 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 12:11:16.622904 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618844 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 12:11:16.622904 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618848 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 12:11:16.622904 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618851 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 12:11:16.622904 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618853 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 12:11:16.622904 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618856 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 12:11:16.622904 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618858 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 12:11:16.622904 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618861 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 12:11:16.622904 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618863 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 12:11:16.622904 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618866 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 12:11:16.622904 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618869 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 12:11:16.623375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618871 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 12:11:16.623375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618874 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 12:11:16.623375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618876 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 12:11:16.623375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618878 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 12:11:16.623375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618881 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 12:11:16.623375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618883 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 12:11:16.623375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618885 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 12:11:16.623375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618888 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 12:11:16.623375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618890 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 12:11:16.623375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618893 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 12:11:16.623375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618895 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 12:11:16.623375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618898 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 12:11:16.623375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618900 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 12:11:16.623375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618902 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 12:11:16.623375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618905 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 12:11:16.623375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618908 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 12:11:16.623375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618910 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 12:11:16.623375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618912 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 12:11:16.623375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618915 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 12:11:16.623375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618918 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 12:11:16.623878 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618923 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 12:11:16.623878 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618926 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 12:11:16.623878 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618928 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 12:11:16.623878 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618931 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 12:11:16.623878 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618933 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 12:11:16.623878 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618936 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 12:11:16.623878 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618938 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 12:11:16.623878 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618941 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 12:11:16.623878 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618946 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 12:11:16.623878 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618949 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 12:11:16.623878 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618952 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 12:11:16.623878 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618955 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 12:11:16.623878 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618958 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 12:11:16.623878 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618961 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 12:11:16.623878 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618963 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 12:11:16.623878 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618966 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 12:11:16.623878 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618968 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 12:11:16.623878 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618970 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 12:11:16.623878 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618973 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 12:11:16.623878 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618976 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.618980 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619047 2556 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619054 2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619060 2556 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619064 2556 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619069 2556 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619072 2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619076 2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619081 2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619084 2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619087 2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619090 2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619095 2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619098 2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619101 2556 flags.go:64] FLAG: --cgroup-root="" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619103 2556 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619106 2556 flags.go:64] FLAG: --client-ca-file="" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619109 2556 flags.go:64] FLAG: --cloud-provider="external" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619112 2556 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619117 2556 flags.go:64] FLAG: --cluster-domain="" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619120 2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619123 2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619126 2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 12:11:16.624347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619129 2556 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619132 2556 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619135 2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619138 2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619141 2556 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619144 2556 flags.go:64] FLAG: --contention-profiling="false" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619147 2556 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619150 2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619153 2556 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619155 2556 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619159 2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619162 2556 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619165 2556 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619167 2556 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619170 2556 flags.go:64] FLAG: --enable-server="true" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619173 2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619177 2556 flags.go:64] FLAG: --event-burst="100" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619180 2556 flags.go:64] FLAG: --event-qps="50" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619183 2556 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619186 2556 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619190 2556 flags.go:64] FLAG: --eviction-hard="" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619195 2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619197 2556 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619200 2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619204 2556 flags.go:64] FLAG: --eviction-soft="" Jul 15 12:11:16.625113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619206 2556 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619209 2556 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619212 2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619216 2556 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619219 2556 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619222 2556 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619224 2556 flags.go:64] FLAG: --feature-gates="" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619228 2556 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619231 2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619234 2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619237 2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619240 2556 flags.go:64] FLAG: --healthz-port="10248" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619243 2556 flags.go:64] FLAG: --help="false" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619245 2556 flags.go:64] FLAG: --hostname-override="ip-10-0-141-45.ec2.internal" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619248 2556 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619251 2556 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619254 2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619257 2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619260 2556 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619263 2556 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619266 2556 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619269 2556 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619271 2556 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 12:11:16.625805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619274 2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619277 2556 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619280 2556 flags.go:64] FLAG: --kube-reserved="" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619283 2556 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619286 2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619290 2556 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619293 2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619296 2556 flags.go:64] FLAG: --lock-file="" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619298 2556 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619301 2556 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619304 2556 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619309 2556 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619312 2556 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619315 2556 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619318 2556 flags.go:64] FLAG: --logging-format="text" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619320 2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619323 2556 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619326 2556 flags.go:64] FLAG: --manifest-url="" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619329 2556 flags.go:64] FLAG: --manifest-url-header="" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619333 2556 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619336 2556 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619340 2556 flags.go:64] FLAG: --max-pods="110" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619343 2556 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619346 2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619348 2556 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 12:11:16.626575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619351 2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619354 2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619357 2556 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619359 2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619367 2556 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619370 2556 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619373 2556 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619375 2556 flags.go:64] FLAG: --pod-cidr="" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619378 2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619383 2556 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619386 2556 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619389 2556 flags.go:64] FLAG: --pods-per-core="0" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619392 2556 flags.go:64] FLAG: --port="10250" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619396 2556 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619399 2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bba20e8a14da4a29" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619402 2556 flags.go:64] FLAG: --qos-reserved="" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619405 2556 flags.go:64] FLAG: --read-only-port="10255" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619408 2556 flags.go:64] FLAG: --register-node="true" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619411 2556 flags.go:64] FLAG: --register-with-taints="" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619414 2556 flags.go:64] FLAG: --registry-burst="10" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619417 2556 flags.go:64] FLAG: --registry-qps="5" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619420 2556 flags.go:64] FLAG: --reserved-cpus="" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619423 2556 flags.go:64] FLAG: --reserved-memory="" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619426 2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619429 2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 12:11:16.627523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619432 2556 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619435 2556 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619437 2556 flags.go:64] FLAG: --runonce="false" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619440 2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619443 2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619446 2556 flags.go:64] FLAG: --seccomp-default="false" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619449 2556 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619451 2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619454 2556 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619457 2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619460 2556 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619462 2556 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619465 2556 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619468 2556 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619470 2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619473 2556 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619478 2556 flags.go:64] FLAG: --system-cgroups="" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619481 2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619486 2556 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619488 2556 flags.go:64] FLAG: --tls-cert-file="" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619492 2556 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619497 2556 flags.go:64] FLAG: --tls-min-version="" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619500 2556 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619502 2556 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619505 2556 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 12:11:16.628156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619508 2556 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619511 2556 flags.go:64] FLAG: --v="2" Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619515 2556 flags.go:64] FLAG: --version="false" Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619519 2556 flags.go:64] FLAG: --vmodule="" Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619523 2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619526 2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619653 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619657 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619660 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619663 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619666 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619669 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619672 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619675 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619678 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619680 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619683 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619685 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619688 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619690 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619693 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 12:11:16.628764 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619695 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 12:11:16.629300 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619698 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 12:11:16.629300 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619702 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 12:11:16.629300 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619704 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 12:11:16.629300 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619707 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 12:11:16.629300 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619709 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 12:11:16.629300 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619713 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 12:11:16.629300 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619716 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 12:11:16.629300 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619718 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 12:11:16.629300 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619721 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 12:11:16.629300 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619723 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 12:11:16.629300 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619725 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 12:11:16.629300 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619728 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 12:11:16.629300 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619730 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 12:11:16.629300 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619733 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 12:11:16.629300 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619735 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 12:11:16.629300 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619738 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 12:11:16.629300 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619742 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 12:11:16.629300 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619746 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 12:11:16.629300 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619749 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 12:11:16.629300 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619752 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 12:11:16.629861 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619755 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 12:11:16.629861 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619757 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 12:11:16.629861 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619760 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 12:11:16.629861 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619762 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 12:11:16.629861 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619765 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 12:11:16.629861 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619767 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 12:11:16.629861 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619788 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 12:11:16.629861 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619792 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 12:11:16.629861 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619796 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 12:11:16.629861 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619801 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 12:11:16.629861 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619804 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 12:11:16.629861 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619807 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 12:11:16.629861 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619809 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 12:11:16.629861 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619813 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 12:11:16.629861 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619816 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 12:11:16.629861 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619819 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 12:11:16.629861 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619821 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 12:11:16.629861 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619825 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 12:11:16.629861 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619827 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 12:11:16.629861 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619830 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 12:11:16.630339 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619832 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 12:11:16.630339 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619836 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 12:11:16.630339 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619840 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 12:11:16.630339 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619842 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 12:11:16.630339 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619845 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 12:11:16.630339 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619848 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 12:11:16.630339 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619850 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 12:11:16.630339 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619853 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 12:11:16.630339 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619857 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 12:11:16.630339 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619860 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 12:11:16.630339 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619864 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 12:11:16.630339 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619866 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 12:11:16.630339 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619869 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 12:11:16.630339 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619872 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 12:11:16.630339 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619875 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 12:11:16.630339 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619877 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 12:11:16.630339 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619880 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 12:11:16.630339 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619883 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 12:11:16.630339 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619886 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 12:11:16.630821 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619888 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 12:11:16.630821 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619891 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 12:11:16.630821 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619893 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 12:11:16.630821 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619896 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 12:11:16.630821 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619898 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 12:11:16.630821 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619901 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 12:11:16.630821 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619904 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 12:11:16.630821 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619907 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 12:11:16.630821 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619910 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 12:11:16.630821 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619912 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 12:11:16.630821 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619916 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 12:11:16.630821 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619919 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 12:11:16.630821 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619921 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 12:11:16.630821 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619923 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 12:11:16.630821 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619926 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 12:11:16.630821 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619928 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 12:11:16.630821 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619931 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 12:11:16.630821 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619933 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 12:11:16.630821 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619935 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 12:11:16.630821 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619938 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 12:11:16.631308 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619941 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 12:11:16.631308 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619943 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 12:11:16.631308 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619946 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 12:11:16.631308 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.619948 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 12:11:16.631308 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.619955 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 12:11:16.631308 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.627831 2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 12:11:16.631308 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.627846 2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 12:11:16.631308 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.628479 2556 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 12:11:16.631308 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.628490 2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 12:11:16.631308 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628544 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 12:11:16.631308 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628548 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 12:11:16.631308 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628552 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 12:11:16.631308 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628555 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 12:11:16.631308 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628557 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 12:11:16.631308 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628560 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 12:11:16.631681 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628564 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 12:11:16.631681 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628566 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 12:11:16.631681 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628569 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 12:11:16.631681 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628572 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 12:11:16.631681 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628575 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 12:11:16.631681 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628578 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 12:11:16.631681 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628581 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 12:11:16.631681 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628584 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 12:11:16.631681 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628587 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 12:11:16.631681 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628590 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 12:11:16.631681 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628592 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 12:11:16.631681 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628595 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 12:11:16.631681 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628598 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 12:11:16.631681 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628600 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 12:11:16.631681 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628602 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 12:11:16.631681 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628605 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 12:11:16.631681 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628607 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 12:11:16.631681 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628610 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 12:11:16.631681 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628613 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 12:11:16.631681 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628615 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 12:11:16.632159 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628617 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 12:11:16.632159 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628620 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 12:11:16.632159 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628622 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 12:11:16.632159 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628625 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 12:11:16.632159 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628628 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 12:11:16.632159 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628631 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 12:11:16.632159 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628633 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 12:11:16.632159 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628636 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 12:11:16.632159 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628639 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 12:11:16.632159 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628641 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 12:11:16.632159 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628645 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 12:11:16.632159 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628649 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 12:11:16.632159 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628651 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 12:11:16.632159 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628654 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 12:11:16.632159 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628658 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 12:11:16.632159 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628663 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 12:11:16.632159 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628666 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 12:11:16.632159 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628669 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 12:11:16.632159 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628672 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 12:11:16.632159 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628674 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 12:11:16.632639 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628678 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 12:11:16.632639 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628680 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 12:11:16.632639 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628683 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 12:11:16.632639 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628685 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 12:11:16.632639 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628688 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 12:11:16.632639 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628690 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 12:11:16.632639 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628693 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 12:11:16.632639 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628696 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 12:11:16.632639 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628698 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 12:11:16.632639 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628700 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 12:11:16.632639 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628703 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 12:11:16.632639 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628705 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 12:11:16.632639 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628708 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 12:11:16.632639 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628710 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 15 12:11:16.632639 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628713 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 12:11:16.632639 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628717 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 12:11:16.632639 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628720 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 12:11:16.632639 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628722 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 12:11:16.632639 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628725 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 12:11:16.633156 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628727 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 12:11:16.633156 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628730 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 12:11:16.633156 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628732 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 12:11:16.633156 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628736 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 12:11:16.633156 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628738 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 12:11:16.633156 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628740 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 12:11:16.633156 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628743 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 12:11:16.633156 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628745 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 12:11:16.633156 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628748 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 12:11:16.633156 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628750 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 12:11:16.633156 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628752 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 12:11:16.633156 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628755 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 12:11:16.633156 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628757 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 12:11:16.633156 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628760 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 12:11:16.633156 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628763 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 12:11:16.633156 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628765 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 12:11:16.633156 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628768 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 12:11:16.633156 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628791 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 12:11:16.633156 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628794 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 12:11:16.633156 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628796 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 12:11:16.633632 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628799 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 12:11:16.633632 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628802 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 12:11:16.633632 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628804 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 12:11:16.633632 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628808 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 12:11:16.633632 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628810 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 12:11:16.633632 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628814 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 12:11:16.633632 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628818 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 12:11:16.633632 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628821 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 12:11:16.633632 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628824 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 12:11:16.633632 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628827 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 12:11:16.633632 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628830 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 12:11:16.633632 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628833 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 12:11:16.633632 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628835 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 12:11:16.633632 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628838 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 12:11:16.634002 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.628844 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 12:11:16.634002 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628959 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 12:11:16.634002 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628965 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 12:11:16.634002 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628968 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 12:11:16.634002 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628971 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 12:11:16.634002 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628974 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 12:11:16.634002 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628977 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 12:11:16.634002 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628980 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 12:11:16.634002 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628983 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 12:11:16.634002 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628985 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 12:11:16.634002 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628988 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 12:11:16.634002 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628990 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 12:11:16.634002 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628994 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 12:11:16.634002 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.628998 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 12:11:16.634002 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629001 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 12:11:16.634365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629003 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 12:11:16.634365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629006 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 12:11:16.634365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629008 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 12:11:16.634365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629011 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 12:11:16.634365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629013 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 12:11:16.634365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629015 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 12:11:16.634365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629018 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 12:11:16.634365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629020 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 12:11:16.634365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629023 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 12:11:16.634365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629025 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 12:11:16.634365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629028 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 12:11:16.634365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629031 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 12:11:16.634365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629034 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 12:11:16.634365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629036 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 12:11:16.634365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629039 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 12:11:16.634365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629041 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 12:11:16.634365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629045 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 12:11:16.634365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629048 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 12:11:16.634365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629051 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 12:11:16.634365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629053 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 12:11:16.634872 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629056 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 12:11:16.634872 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629059 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 12:11:16.634872 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629062 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 12:11:16.634872 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629065 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 12:11:16.634872 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629068 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 12:11:16.634872 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629070 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 12:11:16.634872 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629073 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 12:11:16.634872 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629075 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 12:11:16.634872 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629078 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 12:11:16.634872 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629080 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 12:11:16.634872 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629082 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 12:11:16.634872 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629085 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 12:11:16.634872 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629088 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 12:11:16.634872 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629090 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 12:11:16.634872 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629093 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 12:11:16.634872 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629095 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 12:11:16.634872 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629098 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 12:11:16.634872 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629100 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 12:11:16.634872 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629103 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 12:11:16.634872 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629105 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 12:11:16.635842 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629108 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 12:11:16.635842 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629110 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 12:11:16.635842 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629113 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 12:11:16.635842 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629115 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 12:11:16.635842 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629118 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 12:11:16.635842 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629120 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 12:11:16.635842 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629123 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 12:11:16.635842 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629125 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 12:11:16.635842 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629128 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 12:11:16.635842 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629130 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 12:11:16.635842 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629133 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 12:11:16.635842 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629136 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 12:11:16.635842 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629138 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 12:11:16.635842 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629140 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 12:11:16.635842 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629143 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 12:11:16.635842 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629145 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 12:11:16.635842 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629148 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 12:11:16.635842 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629151 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 12:11:16.635842 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629153 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 12:11:16.636298 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629155 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 12:11:16.636298 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629158 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 12:11:16.636298 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629160 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 12:11:16.636298 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629163 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 12:11:16.636298 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629165 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 12:11:16.636298 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629168 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 12:11:16.636298 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629171 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 12:11:16.636298 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629173 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 12:11:16.636298 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629176 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 12:11:16.636298 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629178 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 12:11:16.636298 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629180 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 12:11:16.636298 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629183 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 12:11:16.636298 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629185 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 12:11:16.636298 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629188 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 12:11:16.636298 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629190 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 12:11:16.636298 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629193 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 12:11:16.636298 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629195 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 12:11:16.636298 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629198 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 12:11:16.636298 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629200 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 12:11:16.636298 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629203 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 12:11:16.636825 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629205 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 12:11:16.636825 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629208 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 12:11:16.636825 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629210 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 12:11:16.636825 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629212 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 12:11:16.636825 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629215 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 12:11:16.636825 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:16.629217 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 15 12:11:16.636825 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.629222 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 12:11:16.636825 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.629320 2556 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 12:11:16.636825 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.631189 2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 12:11:16.636825 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.632222 2556 server.go:1019] "Starting client certificate rotation" Jul 15 12:11:16.636825 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.632320 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 12:11:16.636825 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.632388 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 12:11:16.636825 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.634318 2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 12:11:16.637149 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.636952 2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 12:11:16.648071 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.648048 2556 log.go:25] "Validated CRI v1 runtime API" Jul 15 12:11:16.649972 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.649956 2556 log.go:25] "Validated CRI v1 image API" Jul 15 12:11:16.650764 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.650743 2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 12:11:16.653132 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.653115 2556 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9c462e18-f41e-4853-8bc4-fb7bc767b318:/dev/nvme0n1p4 c2d452f7-3a69-41ae-a613-9563ffd8b1c6:/dev/nvme0n1p3] Jul 15 12:11:16.653193 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.653132 2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 12:11:16.658573 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.658458 2556 manager.go:216] Machine: {Timestamp:2026-07-15 12:11:16.657334782 +0000 UTC m=+0.291729724 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105192 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2906432f2561ad056035b0f2f2b7cb SystemUUID:ec290643-2f25-61ad-0560-35b0f2f2b7cb BootID:4c9a03d2-d3d8-4c7c-be7c-7c7c49003656 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bc:1e:7e:f1:61 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bc:1e:7e:f1:61 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:a1:0f:16:bc:07 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 12:11:16.658573 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.658567 2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 12:11:16.658683 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.658656 2556 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 12:11:16.659177 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.659162 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 12:11:16.661564 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.661542 2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 12:11:16.661719 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.661567 2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-45.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 12:11:16.662170 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.662161 2556 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 12:11:16.662207 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.662172 2556 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 12:11:16.662207 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.662179 2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 12:11:16.662698 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.662689 2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 12:11:16.663827 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.663817 2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 12:11:16.666375 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.666363 2556 state_mem.go:36] "Initialized new in-memory state store" Jul 15 12:11:16.666499 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.666490 2556 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 12:11:16.666528 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.666524 2556 kubelet.go:486] "Attempting to sync node with API server" Jul 15 12:11:16.666555 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.666539 2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 12:11:16.666589 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.666556 2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 12:11:16.666589 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.666573 2556 kubelet.go:398] "Adding apiserver pod source" Jul 15 12:11:16.666589 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.666581 2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 12:11:16.667021 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.667007 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 12:11:16.667021 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.667025 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 12:11:16.670159 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.670078 2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 12:11:16.671715 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.671696 2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 12:11:16.671812 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.671736 2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 12:11:16.671948 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.671935 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 12:11:16.672006 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.671955 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 12:11:16.672006 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.671963 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 12:11:16.672006 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.671972 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 12:11:16.672006 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.671983 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 12:11:16.672006 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.671991 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 12:11:16.672604 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.672589 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 12:11:16.672604 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.672604 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 12:11:16.672710 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.672615 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 12:11:16.672710 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.672622 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 12:11:16.672710 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.672635 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 12:11:16.672710 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.672646 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 12:11:16.673299 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.673287 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 12:11:16.673363 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.673302 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 12:11:16.676406 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.676390 2556 server.go:1268] "Started kubelet" Jul 15 12:11:16.676512 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.676473 2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 12:11:16.676574 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.676495 2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 12:11:16.676614 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.676599 2556 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 12:11:16.677061 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:16.676925 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 12:11:16.677061 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:16.676928 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 12:11:16.677182 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.677037 2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-45.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 12:11:16.677233 ip-10-0-141-45 systemd[1]: Started Kubernetes Kubelet. Jul 15 12:11:16.678616 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.677661 2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 12:11:16.678616 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.678578 2556 server.go:310] "Adding debug handlers to kubelet server" Jul 15 12:11:16.681697 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.681679 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9jcwd" Jul 15 12:11:16.681759 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:16.680903 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-45.ec2.internal.18c2756472dc68be default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-45.ec2.internal,UID:ip-10-0-141-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-45.ec2.internal,},FirstTimestamp:2026-07-15 12:11:16.676364478 +0000 UTC m=+0.310759416,LastTimestamp:2026-07-15 12:11:16.676364478 +0000 UTC m=+0.310759416,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-45.ec2.internal,}" Jul 15 12:11:16.682598 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:16.682573 2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 12:11:16.683991 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.683969 2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 12:11:16.684075 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.683974 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 12:11:16.684206 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:16.684187 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 15 12:11:16.684512 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.684497 2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 12:11:16.684567 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.684501 2556 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 12:11:16.684567 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.684528 2556 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 12:11:16.684690 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.684676 2556 reconstruct.go:97] "Volume reconstruction finished" Jul 15 12:11:16.684690 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.684689 2556 reconciler.go:29] "Reconciler: start to sync state" Jul 15 12:11:16.684989 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.684974 2556 factory.go:55] Registering systemd factory Jul 15 12:11:16.685103 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.685089 2556 factory.go:223] Registration of the systemd container factory successfully Jul 15 12:11:16.685317 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.685302 2556 factory.go:153] Registering CRI-O factory Jul 15 12:11:16.685383 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.685321 2556 factory.go:223] Registration of the crio container factory successfully Jul 15 12:11:16.685445 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.685432 2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 12:11:16.685508 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.685459 2556 factory.go:103] Registering Raw factory Jul 15 12:11:16.685508 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.685474 2556 manager.go:1196] Started watching for new ooms in manager Jul 15 12:11:16.685903 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.685890 2556 manager.go:349] Starting recovery of all containers Jul 15 12:11:16.688059 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:16.688013 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 12:11:16.688295 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:16.688093 2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 15 12:11:16.690399 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.690381 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9jcwd" Jul 15 12:11:16.696512 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.696492 2556 manager.go:354] Recovery completed Jul 15 12:11:16.701957 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.701946 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:11:16.703946 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.703930 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:11:16.704027 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.703963 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:11:16.704027 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.703978 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:11:16.704515 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.704503 2556 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 12:11:16.704515 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.704515 2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 12:11:16.704584 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.704529 2556 state_mem.go:36] "Initialized new in-memory state store" Jul 15 12:11:16.704584 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.704556 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:11:16.705214 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.705201 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:11:16.705277 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.705227 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:11:16.705277 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.705241 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:11:16.705924 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.705905 2556 policy_none.go:49] "None policy: Start" Jul 15 12:11:16.705924 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.705919 2556 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 12:11:16.705982 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.705931 2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 12:11:16.706243 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:16.706172 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-45.ec2.internal.18c2756474814719 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-45.ec2.internal,UID:ip-10-0-141-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-45.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-45.ec2.internal,},FirstTimestamp:2026-07-15 12:11:16.703946521 +0000 UTC m=+0.338341464,LastTimestamp:2026-07-15 12:11:16.703946521 +0000 UTC m=+0.338341464,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-45.ec2.internal,}" Jul 15 12:11:16.707076 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.707063 2556 policy_none.go:47] "Start" Jul 15 12:11:16.753940 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.753923 2556 manager.go:337] "Starting Device Plugin manager" Jul 15 12:11:16.754034 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:16.754008 2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 12:11:16.754034 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.754024 2556 server.go:87] "Starting device plugin registration server" Jul 15 12:11:16.754259 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.754245 2556 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 12:11:16.754314 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.754261 2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 12:11:16.754373 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.754355 2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 12:11:16.754455 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.754440 2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 12:11:16.754455 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.754456 2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 12:11:16.755448 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:16.755433 2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 12:11:16.755513 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:16.755475 2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-45.ec2.internal\" not found" Jul 15 12:11:16.846790 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.846746 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 12:11:16.848031 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.848015 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 12:11:16.848107 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.848049 2556 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 12:11:16.848107 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.848073 2556 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 12:11:16.848107 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:16.848103 2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 12:11:16.850175 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.850157 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 12:11:16.855169 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.855155 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:11:16.855938 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.855925 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:11:16.856003 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.855953 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:11:16.856003 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.855963 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:11:16.856003 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.855984 2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-45.ec2.internal" Jul 15 12:11:16.862870 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.862831 2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-45.ec2.internal" Jul 15 12:11:16.862870 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:16.862852 2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-45.ec2.internal\": node \"ip-10-0-141-45.ec2.internal\" not found" Jul 15 12:11:16.877201 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:16.877177 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 15 12:11:16.949028 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.949001 2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal"] Jul 15 12:11:16.949128 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.949073 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:11:16.949931 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.949916 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:11:16.950010 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.949952 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:11:16.950010 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.949963 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:11:16.951596 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.951583 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:11:16.951725 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.951700 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" Jul 15 12:11:16.951763 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.951733 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:11:16.952428 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.952411 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:11:16.952536 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.952437 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:11:16.952536 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.952437 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:11:16.952536 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.952447 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:11:16.952536 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.952462 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:11:16.952536 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.952475 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:11:16.953847 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.953833 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" Jul 15 12:11:16.953925 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.953860 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:11:16.954578 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.954563 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:11:16.954666 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.954587 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:11:16.954666 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.954596 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:11:16.977911 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:16.977890 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 15 12:11:16.986195 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.986178 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1c7bc3b4c133f1b0578384be47f8d38-config\") pod \"kube-apiserver-proxy-ip-10-0-141-45.ec2.internal\" (UID: \"a1c7bc3b4c133f1b0578384be47f8d38\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" Jul 15 12:11:16.986255 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.986202 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/34b43d5f526b7974ac738b5aaab0d959-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"34b43d5f526b7974ac738b5aaab0d959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" Jul 15 12:11:16.986255 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:16.986219 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34b43d5f526b7974ac738b5aaab0d959-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"34b43d5f526b7974ac738b5aaab0d959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" Jul 15 12:11:16.989324 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:16.989310 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-45.ec2.internal\" not found" node="ip-10-0-141-45.ec2.internal" Jul 15 12:11:16.993696 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:16.993679 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-45.ec2.internal\" not found" node="ip-10-0-141-45.ec2.internal" Jul 15 12:11:17.078423 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:17.078391 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 15 12:11:17.086726 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.086703 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1c7bc3b4c133f1b0578384be47f8d38-config\") pod \"kube-apiserver-proxy-ip-10-0-141-45.ec2.internal\" (UID: \"a1c7bc3b4c133f1b0578384be47f8d38\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" Jul 15 12:11:17.086820 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.086737 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/34b43d5f526b7974ac738b5aaab0d959-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"34b43d5f526b7974ac738b5aaab0d959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" Jul 15 12:11:17.086820 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.086760 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34b43d5f526b7974ac738b5aaab0d959-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"34b43d5f526b7974ac738b5aaab0d959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" Jul 15 12:11:17.086925 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.086824 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1c7bc3b4c133f1b0578384be47f8d38-config\") pod \"kube-apiserver-proxy-ip-10-0-141-45.ec2.internal\" (UID: \"a1c7bc3b4c133f1b0578384be47f8d38\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" Jul 15 12:11:17.086925 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.086838 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/34b43d5f526b7974ac738b5aaab0d959-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"34b43d5f526b7974ac738b5aaab0d959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" Jul 15 12:11:17.086925 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.086829 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34b43d5f526b7974ac738b5aaab0d959-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"34b43d5f526b7974ac738b5aaab0d959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" Jul 15 12:11:17.179216 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:17.179149 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 15 12:11:17.279697 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:17.279669 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 15 12:11:17.291165 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.291138 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" Jul 15 12:11:17.296761 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.296743 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" Jul 15 12:11:17.380086 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:17.380060 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 15 12:11:17.480669 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:17.480604 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 15 12:11:17.492837 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.492817 2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 12:11:17.538957 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.538934 2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 12:11:17.584303 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.584285 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" Jul 15 12:11:17.593025 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.592997 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 12:11:17.594515 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.594502 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" Jul 15 12:11:17.604109 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.604093 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 12:11:17.632271 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.632235 2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 12:11:17.632857 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.632352 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 12:11:17.632857 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.632362 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 12:11:17.632857 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.632359 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 12:11:17.667486 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.667470 2556 apiserver.go:52] "Watching apiserver" Jul 15 12:11:17.683302 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.683284 2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 12:11:17.683610 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.683591 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-4crgq","openshift-ovn-kubernetes/ovnkube-node-5fdr4","kube-system/konnectivity-agent-4zpzj","kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x","openshift-cluster-node-tuning-operator/tuned-ndgsc","openshift-dns/node-resolver-b8nd7","openshift-image-registry/node-ca-p8nqm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal","openshift-multus/multus-8ngfj","kube-system/global-pull-secret-syncer-sj4hd","openshift-multus/multus-additional-cni-plugins-k2z7d","openshift-multus/network-metrics-daemon-kfbnz","openshift-network-diagnostics/network-check-target-hqq7p"] Jul 15 12:11:17.684113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.684094 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 12:11:17.685008 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.684994 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.686013 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.685998 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:17.686108 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:17.686079 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfbnz" podUID="e79deb5e-778c-420d-a65d-671a7d3d28db" Jul 15 12:11:17.687093 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.687075 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4zpzj" Jul 15 12:11:17.687446 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.687431 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 12:11:17.687649 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.687629 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 12:11:17.687649 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.687640 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8djs7\"" Jul 15 12:11:17.687869 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.687853 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 12:11:17.687936 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.687861 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 12:11:17.688263 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.688249 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:17.688326 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:17.688310 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqq7p" podUID="ad5f3fa8-e4d3-42bf-8b16-94961009814d" Jul 15 12:11:17.689516 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.689504 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.689702 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.689622 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjzk7\" (UniqueName: \"kubernetes.io/projected/e79deb5e-778c-420d-a65d-671a7d3d28db-kube-api-access-rjzk7\") pod \"network-metrics-daemon-kfbnz\" (UID: \"e79deb5e-778c-420d-a65d-671a7d3d28db\") " pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:17.689702 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.689659 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-system-cni-dir\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.689702 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.689687 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vsb9j\"" Jul 15 12:11:17.689899 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.689686 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-cnibin\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.689899 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.689746 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-os-release\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.689899 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.689816 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-multus-socket-dir-parent\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.689899 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.689845 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-host-run-netns\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.689899 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.689870 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-hostroot\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.689899 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.689893 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6132cce3-5328-44d3-af00-a59e66e03f62-multus-daemon-config\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.690147 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.689914 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-multus-cni-dir\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.690147 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.689938 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-host-var-lib-cni-multus\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.690147 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.689960 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-host-var-lib-kubelet\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.690147 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.689984 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-etc-kubernetes\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.690147 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.689990 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 12:11:17.690147 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.690024 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-host-var-lib-cni-bin\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.690147 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.690048 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6132cce3-5328-44d3-af00-a59e66e03f62-cni-binary-copy\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.690147 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.690071 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdpkn\" (UniqueName: \"kubernetes.io/projected/6132cce3-5328-44d3-af00-a59e66e03f62-kube-api-access-sdpkn\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.690147 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.690097 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs\") pod \"network-metrics-daemon-kfbnz\" (UID: \"e79deb5e-778c-420d-a65d-671a7d3d28db\") " pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:17.690147 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.690133 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-host-run-k8s-cni-cncf-io\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.690147 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.690151 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-multus-conf-dir\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.690478 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.690175 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-host-run-multus-certs\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.690478 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.690212 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 12:11:17.690667 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.690652 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.691739 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.691724 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p8nqm" Jul 15 12:11:17.691877 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.691844 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 12:11:17.691973 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.691955 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w5mcp\"" Jul 15 12:11:17.692097 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.692082 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 12:11:17.692145 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.692134 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 12:11:17.692748 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.692735 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 12:11:17.692933 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.692920 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sb2lp\"" Jul 15 12:11:17.693095 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.693081 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 12:11:17.693159 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.693129 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b8nd7" Jul 15 12:11:17.694271 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.694255 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gnk7\"" Jul 15 12:11:17.694494 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.694480 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 12:11:17.694583 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.694516 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 12:11:17.694636 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.694602 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 12:11:17.694683 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.694672 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.695200 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.695185 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gj684\"" Jul 15 12:11:17.695513 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.695499 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 12:11:17.695573 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.695526 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 12:11:17.696151 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.696134 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4crgq" Jul 15 12:11:17.696548 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.696524 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 12:06:16 +0000 UTC" deadline="2028-03-16 00:52:06.712256628 +0000 UTC" Jul 15 12:11:17.696548 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.696545 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14628h40m49.015714041s" Jul 15 12:11:17.696645 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.696622 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 12:11:17.696925 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.696910 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 12:11:17.697137 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.697123 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 12:11:17.697217 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.697197 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 12:11:17.697289 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.697275 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tg8dg\"" Jul 15 12:11:17.697659 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.697644 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:17.697737 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:17.697718 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4hd" podUID="24a77a8a-6ec5-4558-8ce8-96ae7f4c9547" Jul 15 12:11:17.698757 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.698493 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 12:11:17.698757 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.698600 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 12:11:17.698757 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.698654 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 12:11:17.698757 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.698604 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 12:11:17.698757 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.698669 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-txlml\"" Jul 15 12:11:17.698757 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.698601 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 12:11:17.698757 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.698656 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 12:11:17.699153 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.698989 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.701523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.701503 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 12:11:17.701749 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.701734 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xgn9m\"" Jul 15 12:11:17.701824 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.701793 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 12:11:17.716728 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.716713 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-45xdx" Jul 15 12:11:17.724393 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.724379 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-45xdx" Jul 15 12:11:17.785498 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.785481 2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 12:11:17.791207 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791190 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.791272 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791215 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44f10944-d216-4dc4-8455-7460d9849743-device-dir\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.791272 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791231 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f249n\" (UniqueName: \"kubernetes.io/projected/44f10944-d216-4dc4-8455-7460d9849743-kube-api-access-f249n\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.791272 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791247 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e-tmp-dir\") pod \"node-resolver-b8nd7\" (UID: \"a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e\") " pod="openshift-dns/node-resolver-b8nd7" Jul 15 12:11:17.791272 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791265 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret\") pod \"global-pull-secret-syncer-sj4hd\" (UID: \"24a77a8a-6ec5-4558-8ce8-96ae7f4c9547\") " pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:17.791398 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791297 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-lib-modules\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.791398 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791324 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-cnibin\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.791398 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791338 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-os-release\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.791398 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791372 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-etc-kubernetes\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.791398 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791396 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-os-release\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.791578 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791401 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44f10944-d216-4dc4-8455-7460d9849743-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.791578 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791418 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-system-cni-dir\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.791578 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791437 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.791578 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791442 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-cnibin\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.791578 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791452 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27881d83-289d-4a88-beba-585f786c0e0c-etc-tuned\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.791578 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791454 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-etc-kubernetes\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.791578 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791479 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44f10944-d216-4dc4-8455-7460d9849743-socket-dir\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.791578 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791501 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44f10944-d216-4dc4-8455-7460d9849743-registration-dir\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.791578 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791515 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/da745f4b-00f5-4b21-91f3-64907b52ebea-agent-certs\") pod \"konnectivity-agent-4zpzj\" (UID: \"da745f4b-00f5-4b21-91f3-64907b52ebea\") " pod="kube-system/konnectivity-agent-4zpzj" Jul 15 12:11:17.791578 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791528 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-etc-kubernetes\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.791578 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791567 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-run-systemd\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.791903 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791588 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-run-openvswitch\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.791903 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791606 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a434baae-76d0-466a-b7e5-7e0e1e57d90e-env-overrides\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.791903 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791637 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6132cce3-5328-44d3-af00-a59e66e03f62-cni-binary-copy\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.791903 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791688 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs\") pod \"network-metrics-daemon-kfbnz\" (UID: \"e79deb5e-778c-420d-a65d-671a7d3d28db\") " pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:17.791903 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791709 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-dbus\") pod \"global-pull-secret-syncer-sj4hd\" (UID: \"24a77a8a-6ec5-4558-8ce8-96ae7f4c9547\") " pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:17.791903 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791738 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-cni-binary-copy\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.791903 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791754 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/da745f4b-00f5-4b21-91f3-64907b52ebea-konnectivity-ca\") pod \"konnectivity-agent-4zpzj\" (UID: \"da745f4b-00f5-4b21-91f3-64907b52ebea\") " pod="kube-system/konnectivity-agent-4zpzj" Jul 15 12:11:17.791903 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791803 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-node-log\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.791903 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791826 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e-hosts-file\") pod \"node-resolver-b8nd7\" (UID: \"a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e\") " pod="openshift-dns/node-resolver-b8nd7" Jul 15 12:11:17.791903 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:17.791838 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:11:17.792172 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791840 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f476f12b-f4b6-4c20-b26c-2c88eac44e2d-host-slash\") pod \"iptables-alerter-4crgq\" (UID: \"f476f12b-f4b6-4c20-b26c-2c88eac44e2d\") " pod="openshift-network-operator/iptables-alerter-4crgq" Jul 15 12:11:17.792172 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:17.791921 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs podName:e79deb5e-778c-420d-a65d-671a7d3d28db nodeName:}" failed. No retries permitted until 2026-07-15 12:11:18.291888791 +0000 UTC m=+1.926283738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs") pod "network-metrics-daemon-kfbnz" (UID: "e79deb5e-778c-420d-a65d-671a7d3d28db") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:11:17.792172 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791949 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-os-release\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.792172 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791975 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.792172 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.791998 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8-serviceca\") pod \"node-ca-p8nqm\" (UID: \"6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8\") " pod="openshift-image-registry/node-ca-p8nqm" Jul 15 12:11:17.792172 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792014 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-run-netns\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.792172 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792030 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-host-run-netns\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.792172 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792044 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-etc-openvswitch\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.792172 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792077 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-host-run-netns\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.792172 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792095 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-cni-netd\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.792172 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792112 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-host-var-lib-kubelet\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.792172 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792131 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27881d83-289d-4a88-beba-585f786c0e0c-tmp\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.792172 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792145 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-slash\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.792172 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792160 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-run-ovn-kubernetes\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.792172 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792174 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqmtr\" (UniqueName: \"kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr\") pod \"network-check-target-hqq7p\" (UID: \"ad5f3fa8-e4d3-42bf-8b16-94961009814d\") " pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:17.792575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792181 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-host-var-lib-kubelet\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.792575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792205 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7nps\" (UniqueName: \"kubernetes.io/projected/27881d83-289d-4a88-beba-585f786c0e0c-kube-api-access-x7nps\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.792575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792241 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-cni-bin\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.792575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792260 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a434baae-76d0-466a-b7e5-7e0e1e57d90e-ovnkube-script-lib\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.792575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792283 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-multus-conf-dir\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.792575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792306 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44f10944-d216-4dc4-8455-7460d9849743-etc-selinux\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.792575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792330 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxv48\" (UniqueName: \"kubernetes.io/projected/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-kube-api-access-pxv48\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.792575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792349 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-etc-sysconfig\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.792575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792350 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-multus-conf-dir\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.792575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792362 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-etc-sysctl-d\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.792575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792377 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-sys\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.792575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792391 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8-host\") pod \"node-ca-p8nqm\" (UID: \"6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8\") " pod="openshift-image-registry/node-ca-p8nqm" Jul 15 12:11:17.792575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792414 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5blrr\" (UniqueName: \"kubernetes.io/projected/6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8-kube-api-access-5blrr\") pod \"node-ca-p8nqm\" (UID: \"6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8\") " pod="openshift-image-registry/node-ca-p8nqm" Jul 15 12:11:17.792575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792435 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-hostroot\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.792575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792460 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-cnibin\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.792575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792484 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-run\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.792575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792484 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-hostroot\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.793072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792510 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-host\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.793072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792537 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-kubelet\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.793072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792551 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-run-ovn\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.793072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792568 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-host-var-lib-cni-multus\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.793072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792600 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv7z7\" (UniqueName: \"kubernetes.io/projected/f476f12b-f4b6-4c20-b26c-2c88eac44e2d-kube-api-access-zv7z7\") pod \"iptables-alerter-4crgq\" (UID: \"f476f12b-f4b6-4c20-b26c-2c88eac44e2d\") " pod="openshift-network-operator/iptables-alerter-4crgq" Jul 15 12:11:17.793072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792614 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-etc-systemd\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.793072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792629 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-var-lib-openvswitch\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.793072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792653 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdpkn\" (UniqueName: \"kubernetes.io/projected/6132cce3-5328-44d3-af00-a59e66e03f62-kube-api-access-sdpkn\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.793072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792674 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-host-var-lib-cni-multus\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.793072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792677 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.793072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792719 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-var-lib-kubelet\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.793072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792746 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-systemd-units\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.793072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792761 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-host-run-k8s-cni-cncf-io\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.793072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792797 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-host-run-multus-certs\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.793072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792813 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-host-run-k8s-cni-cncf-io\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.793072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792818 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjzk7\" (UniqueName: \"kubernetes.io/projected/e79deb5e-778c-420d-a65d-671a7d3d28db-kube-api-access-rjzk7\") pod \"network-metrics-daemon-kfbnz\" (UID: \"e79deb5e-778c-420d-a65d-671a7d3d28db\") " pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:17.793072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792834 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-etc-modprobe-d\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.793533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792838 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-host-run-multus-certs\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.793533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792848 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a434baae-76d0-466a-b7e5-7e0e1e57d90e-ovnkube-config\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.793533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792870 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-system-cni-dir\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.793533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792901 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-multus-socket-dir-parent\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.793533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792917 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6132cce3-5328-44d3-af00-a59e66e03f62-multus-daemon-config\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.793533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792935 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-kubelet-config\") pod \"global-pull-secret-syncer-sj4hd\" (UID: \"24a77a8a-6ec5-4558-8ce8-96ae7f4c9547\") " pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:17.793533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792963 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-etc-sysctl-conf\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.793533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792985 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-multus-socket-dir-parent\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.793533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.792994 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-multus-cni-dir\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.793533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.793016 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44f10944-d216-4dc4-8455-7460d9849743-sys-fs\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.793533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.793038 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swjff\" (UniqueName: \"kubernetes.io/projected/a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e-kube-api-access-swjff\") pod \"node-resolver-b8nd7\" (UID: \"a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e\") " pod="openshift-dns/node-resolver-b8nd7" Jul 15 12:11:17.793533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.793060 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f476f12b-f4b6-4c20-b26c-2c88eac44e2d-iptables-alerter-script\") pod \"iptables-alerter-4crgq\" (UID: \"f476f12b-f4b6-4c20-b26c-2c88eac44e2d\") " pod="openshift-network-operator/iptables-alerter-4crgq" Jul 15 12:11:17.793533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.793061 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-system-cni-dir\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.793533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.793075 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a434baae-76d0-466a-b7e5-7e0e1e57d90e-ovn-node-metrics-cert\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.793533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.793062 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-multus-cni-dir\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.793533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.793092 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h669j\" (UniqueName: \"kubernetes.io/projected/a434baae-76d0-466a-b7e5-7e0e1e57d90e-kube-api-access-h669j\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.793533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.793110 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-host-var-lib-cni-bin\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.794006 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.793124 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-log-socket\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.794006 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.793154 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6132cce3-5328-44d3-af00-a59e66e03f62-host-var-lib-cni-bin\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.794006 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.793209 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6132cce3-5328-44d3-af00-a59e66e03f62-cni-binary-copy\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.794006 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.793311 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6132cce3-5328-44d3-af00-a59e66e03f62-multus-daemon-config\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.800549 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.800524 2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 12:11:17.803339 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.803320 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdpkn\" (UniqueName: \"kubernetes.io/projected/6132cce3-5328-44d3-af00-a59e66e03f62-kube-api-access-sdpkn\") pod \"multus-8ngfj\" (UID: \"6132cce3-5328-44d3-af00-a59e66e03f62\") " pod="openshift-multus/multus-8ngfj" Jul 15 12:11:17.803436 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.803363 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjzk7\" (UniqueName: \"kubernetes.io/projected/e79deb5e-778c-420d-a65d-671a7d3d28db-kube-api-access-rjzk7\") pod \"network-metrics-daemon-kfbnz\" (UID: \"e79deb5e-778c-420d-a65d-671a7d3d28db\") " pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:17.893701 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.893671 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44f10944-d216-4dc4-8455-7460d9849743-socket-dir\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.893701 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.893698 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44f10944-d216-4dc4-8455-7460d9849743-registration-dir\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.893897 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.893712 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/da745f4b-00f5-4b21-91f3-64907b52ebea-agent-certs\") pod \"konnectivity-agent-4zpzj\" (UID: \"da745f4b-00f5-4b21-91f3-64907b52ebea\") " pod="kube-system/konnectivity-agent-4zpzj" Jul 15 12:11:17.893897 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.893724 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-etc-kubernetes\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.893897 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.893744 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-run-systemd\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.893897 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.893803 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44f10944-d216-4dc4-8455-7460d9849743-registration-dir\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.893897 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.893808 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-run-systemd\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.893897 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.893822 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-run-openvswitch\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.893897 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.893839 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-etc-kubernetes\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.893897 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.893849 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a434baae-76d0-466a-b7e5-7e0e1e57d90e-env-overrides\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.893897 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.893872 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44f10944-d216-4dc4-8455-7460d9849743-socket-dir\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.893897 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.893848 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-run-openvswitch\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.894300 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.893905 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-dbus\") pod \"global-pull-secret-syncer-sj4hd\" (UID: \"24a77a8a-6ec5-4558-8ce8-96ae7f4c9547\") " pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:17.894300 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.893927 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-cni-binary-copy\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.894300 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.893942 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/da745f4b-00f5-4b21-91f3-64907b52ebea-konnectivity-ca\") pod \"konnectivity-agent-4zpzj\" (UID: \"da745f4b-00f5-4b21-91f3-64907b52ebea\") " pod="kube-system/konnectivity-agent-4zpzj" Jul 15 12:11:17.894300 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.893954 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-node-log\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.894300 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.893969 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e-hosts-file\") pod \"node-resolver-b8nd7\" (UID: \"a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e\") " pod="openshift-dns/node-resolver-b8nd7" Jul 15 12:11:17.894300 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.893992 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f476f12b-f4b6-4c20-b26c-2c88eac44e2d-host-slash\") pod \"iptables-alerter-4crgq\" (UID: \"f476f12b-f4b6-4c20-b26c-2c88eac44e2d\") " pod="openshift-network-operator/iptables-alerter-4crgq" Jul 15 12:11:17.894300 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894011 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-os-release\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.894300 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894033 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.894300 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894030 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-node-log\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.894300 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894052 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8-serviceca\") pod \"node-ca-p8nqm\" (UID: \"6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8\") " pod="openshift-image-registry/node-ca-p8nqm" Jul 15 12:11:17.894300 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894056 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e-hosts-file\") pod \"node-resolver-b8nd7\" (UID: \"a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e\") " pod="openshift-dns/node-resolver-b8nd7" Jul 15 12:11:17.894300 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894072 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-run-netns\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.894300 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894094 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-etc-openvswitch\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.894300 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894103 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-os-release\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.894300 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894117 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-cni-netd\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.894300 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894143 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27881d83-289d-4a88-beba-585f786c0e0c-tmp\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.894300 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894151 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-run-netns\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.895082 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894163 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-slash\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.895082 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894116 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-dbus\") pod \"global-pull-secret-syncer-sj4hd\" (UID: \"24a77a8a-6ec5-4558-8ce8-96ae7f4c9547\") " pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:17.895082 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894186 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-run-ovn-kubernetes\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.895082 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894198 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-etc-openvswitch\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.895082 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894199 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.895082 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894209 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqmtr\" (UniqueName: \"kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr\") pod \"network-check-target-hqq7p\" (UID: \"ad5f3fa8-e4d3-42bf-8b16-94961009814d\") " pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:17.895082 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894228 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a434baae-76d0-466a-b7e5-7e0e1e57d90e-env-overrides\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.895082 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894235 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-slash\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.895082 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894242 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-cni-netd\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.895082 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894144 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f476f12b-f4b6-4c20-b26c-2c88eac44e2d-host-slash\") pod \"iptables-alerter-4crgq\" (UID: \"f476f12b-f4b6-4c20-b26c-2c88eac44e2d\") " pod="openshift-network-operator/iptables-alerter-4crgq" Jul 15 12:11:17.895082 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894266 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7nps\" (UniqueName: \"kubernetes.io/projected/27881d83-289d-4a88-beba-585f786c0e0c-kube-api-access-x7nps\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.895082 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894278 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-run-ovn-kubernetes\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.895082 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894288 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-cni-bin\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.895082 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894312 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a434baae-76d0-466a-b7e5-7e0e1e57d90e-ovnkube-script-lib\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.895082 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894334 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44f10944-d216-4dc4-8455-7460d9849743-etc-selinux\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.895082 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894356 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxv48\" (UniqueName: \"kubernetes.io/projected/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-kube-api-access-pxv48\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.895082 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894378 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-etc-sysconfig\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.895809 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894400 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-etc-sysctl-d\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.895809 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894421 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-sys\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.895809 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894440 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8-host\") pod \"node-ca-p8nqm\" (UID: \"6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8\") " pod="openshift-image-registry/node-ca-p8nqm" Jul 15 12:11:17.895809 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894377 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-cni-bin\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.895809 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894468 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5blrr\" (UniqueName: \"kubernetes.io/projected/6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8-kube-api-access-5blrr\") pod \"node-ca-p8nqm\" (UID: \"6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8\") " pod="openshift-image-registry/node-ca-p8nqm" Jul 15 12:11:17.895809 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894491 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-cnibin\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.895809 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894512 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-run\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.895809 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894531 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-host\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.895809 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894552 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-kubelet\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.895809 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894562 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8-serviceca\") pod \"node-ca-p8nqm\" (UID: \"6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8\") " pod="openshift-image-registry/node-ca-p8nqm" Jul 15 12:11:17.895809 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894593 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-run-ovn\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.895809 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894625 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv7z7\" (UniqueName: \"kubernetes.io/projected/f476f12b-f4b6-4c20-b26c-2c88eac44e2d-kube-api-access-zv7z7\") pod \"iptables-alerter-4crgq\" (UID: \"f476f12b-f4b6-4c20-b26c-2c88eac44e2d\") " pod="openshift-network-operator/iptables-alerter-4crgq" Jul 15 12:11:17.895809 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894644 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-etc-sysconfig\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.895809 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894646 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-etc-systemd\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.895809 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894684 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-var-lib-openvswitch\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.895809 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894698 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-etc-systemd\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.895809 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894727 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.895809 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894740 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-cnibin\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.896806 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894753 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-var-lib-kubelet\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.896806 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894799 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-systemd-units\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.896806 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894818 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-run\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.896806 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894830 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-etc-modprobe-d\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.896806 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894852 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a434baae-76d0-466a-b7e5-7e0e1e57d90e-ovnkube-config\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.896806 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894865 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-host\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.896806 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894873 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-sys\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.896806 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894878 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-kubelet-config\") pod \"global-pull-secret-syncer-sj4hd\" (UID: \"24a77a8a-6ec5-4558-8ce8-96ae7f4c9547\") " pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:17.896806 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894905 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-etc-sysctl-conf\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.896806 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894440 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/da745f4b-00f5-4b21-91f3-64907b52ebea-konnectivity-ca\") pod \"konnectivity-agent-4zpzj\" (UID: \"da745f4b-00f5-4b21-91f3-64907b52ebea\") " pod="kube-system/konnectivity-agent-4zpzj" Jul 15 12:11:17.896806 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894929 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44f10944-d216-4dc4-8455-7460d9849743-sys-fs\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.896806 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894944 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44f10944-d216-4dc4-8455-7460d9849743-etc-selinux\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.896806 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894951 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swjff\" (UniqueName: \"kubernetes.io/projected/a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e-kube-api-access-swjff\") pod \"node-resolver-b8nd7\" (UID: \"a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e\") " pod="openshift-dns/node-resolver-b8nd7" Jul 15 12:11:17.896806 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894958 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-etc-sysctl-d\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.896806 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894974 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f476f12b-f4b6-4c20-b26c-2c88eac44e2d-iptables-alerter-script\") pod \"iptables-alerter-4crgq\" (UID: \"f476f12b-f4b6-4c20-b26c-2c88eac44e2d\") " pod="openshift-network-operator/iptables-alerter-4crgq" Jul 15 12:11:17.896806 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894996 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a434baae-76d0-466a-b7e5-7e0e1e57d90e-ovn-node-metrics-cert\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.896806 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895009 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a434baae-76d0-466a-b7e5-7e0e1e57d90e-ovnkube-script-lib\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.897542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895015 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-run-ovn\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.897542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895018 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h669j\" (UniqueName: \"kubernetes.io/projected/a434baae-76d0-466a-b7e5-7e0e1e57d90e-kube-api-access-h669j\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.897542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895046 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-kubelet\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.897542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895086 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-log-socket\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.897542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895064 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-var-lib-openvswitch\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.897542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895058 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-log-socket\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.897542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.894908 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8-host\") pod \"node-ca-p8nqm\" (UID: \"6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8\") " pod="openshift-image-registry/node-ca-p8nqm" Jul 15 12:11:17.897542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895170 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-etc-modprobe-d\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.897542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895266 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.897542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895291 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44f10944-d216-4dc4-8455-7460d9849743-sys-fs\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.897542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895297 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44f10944-d216-4dc4-8455-7460d9849743-device-dir\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.897542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895286 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.897542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895337 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-cni-binary-copy\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.897542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895339 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a434baae-76d0-466a-b7e5-7e0e1e57d90e-systemd-units\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.897542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895395 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f249n\" (UniqueName: \"kubernetes.io/projected/44f10944-d216-4dc4-8455-7460d9849743-kube-api-access-f249n\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.897542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895402 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-etc-sysctl-conf\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.897542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895417 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e-tmp-dir\") pod \"node-resolver-b8nd7\" (UID: \"a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e\") " pod="openshift-dns/node-resolver-b8nd7" Jul 15 12:11:17.898232 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895438 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-var-lib-kubelet\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.898232 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895441 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret\") pod \"global-pull-secret-syncer-sj4hd\" (UID: \"24a77a8a-6ec5-4558-8ce8-96ae7f4c9547\") " pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:17.898232 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895514 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.898232 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895569 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-lib-modules\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.898232 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895611 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-kubelet-config\") pod \"global-pull-secret-syncer-sj4hd\" (UID: \"24a77a8a-6ec5-4558-8ce8-96ae7f4c9547\") " pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:17.898232 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895620 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44f10944-d216-4dc4-8455-7460d9849743-device-dir\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.898232 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:17.895674 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:11:17.898232 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895701 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a434baae-76d0-466a-b7e5-7e0e1e57d90e-ovnkube-config\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.898232 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:17.895722 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret podName:24a77a8a-6ec5-4558-8ce8-96ae7f4c9547 nodeName:}" failed. No retries permitted until 2026-07-15 12:11:18.395705146 +0000 UTC m=+2.030100070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret") pod "global-pull-secret-syncer-sj4hd" (UID: "24a77a8a-6ec5-4558-8ce8-96ae7f4c9547") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:11:17.898232 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895811 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44f10944-d216-4dc4-8455-7460d9849743-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.898232 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895818 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27881d83-289d-4a88-beba-585f786c0e0c-lib-modules\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.898232 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895841 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-system-cni-dir\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.898232 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895875 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f476f12b-f4b6-4c20-b26c-2c88eac44e2d-iptables-alerter-script\") pod \"iptables-alerter-4crgq\" (UID: \"f476f12b-f4b6-4c20-b26c-2c88eac44e2d\") " pod="openshift-network-operator/iptables-alerter-4crgq" Jul 15 12:11:17.898232 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895888 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.898232 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895910 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27881d83-289d-4a88-beba-585f786c0e0c-etc-tuned\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.898232 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.895926 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-system-cni-dir\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.898232 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.896063 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e-tmp-dir\") pod \"node-resolver-b8nd7\" (UID: \"a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e\") " pod="openshift-dns/node-resolver-b8nd7" Jul 15 12:11:17.898658 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.896124 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44f10944-d216-4dc4-8455-7460d9849743-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.898658 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.896262 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:17.898658 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.896670 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/da745f4b-00f5-4b21-91f3-64907b52ebea-agent-certs\") pod \"konnectivity-agent-4zpzj\" (UID: \"da745f4b-00f5-4b21-91f3-64907b52ebea\") " pod="kube-system/konnectivity-agent-4zpzj" Jul 15 12:11:17.898658 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.897202 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27881d83-289d-4a88-beba-585f786c0e0c-tmp\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.898658 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.897873 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27881d83-289d-4a88-beba-585f786c0e0c-etc-tuned\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.898658 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.898200 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a434baae-76d0-466a-b7e5-7e0e1e57d90e-ovn-node-metrics-cert\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.903121 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:17.903093 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:11:17.903121 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:17.903123 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:11:17.903309 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:17.903135 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mqmtr for pod openshift-network-diagnostics/network-check-target-hqq7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:11:17.903309 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:17.903253 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr podName:ad5f3fa8-e4d3-42bf-8b16-94961009814d nodeName:}" failed. No retries permitted until 2026-07-15 12:11:18.403236603 +0000 UTC m=+2.037631540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mqmtr" (UniqueName: "kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr") pod "network-check-target-hqq7p" (UID: "ad5f3fa8-e4d3-42bf-8b16-94961009814d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:11:17.904603 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.904572 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv7z7\" (UniqueName: \"kubernetes.io/projected/f476f12b-f4b6-4c20-b26c-2c88eac44e2d-kube-api-access-zv7z7\") pod \"iptables-alerter-4crgq\" (UID: \"f476f12b-f4b6-4c20-b26c-2c88eac44e2d\") " pod="openshift-network-operator/iptables-alerter-4crgq" Jul 15 12:11:17.904804 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.904787 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5blrr\" (UniqueName: \"kubernetes.io/projected/6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8-kube-api-access-5blrr\") pod \"node-ca-p8nqm\" (UID: \"6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8\") " pod="openshift-image-registry/node-ca-p8nqm" Jul 15 12:11:17.905168 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.905148 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h669j\" (UniqueName: \"kubernetes.io/projected/a434baae-76d0-466a-b7e5-7e0e1e57d90e-kube-api-access-h669j\") pod \"ovnkube-node-5fdr4\" (UID: \"a434baae-76d0-466a-b7e5-7e0e1e57d90e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:17.905406 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.905389 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swjff\" (UniqueName: \"kubernetes.io/projected/a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e-kube-api-access-swjff\") pod \"node-resolver-b8nd7\" (UID: \"a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e\") " pod="openshift-dns/node-resolver-b8nd7" Jul 15 12:11:17.905575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.905555 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f249n\" (UniqueName: \"kubernetes.io/projected/44f10944-d216-4dc4-8455-7460d9849743-kube-api-access-f249n\") pod \"aws-ebs-csi-driver-node-5jw5x\" (UID: \"44f10944-d216-4dc4-8455-7460d9849743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:17.905741 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.905724 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7nps\" (UniqueName: \"kubernetes.io/projected/27881d83-289d-4a88-beba-585f786c0e0c-kube-api-access-x7nps\") pod \"tuned-ndgsc\" (UID: \"27881d83-289d-4a88-beba-585f786c0e0c\") " pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:17.906086 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:17.906068 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxv48\" (UniqueName: \"kubernetes.io/projected/757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8-kube-api-access-pxv48\") pod \"multus-additional-cni-plugins-k2z7d\" (UID: \"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8\") " pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:18.006721 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.006646 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8ngfj" Jul 15 12:11:18.019119 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.019095 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4zpzj" Jul 15 12:11:18.033053 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.033032 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" Jul 15 12:11:18.039431 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.039416 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" Jul 15 12:11:18.065844 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.065821 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p8nqm" Jul 15 12:11:18.069221 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.069206 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b8nd7" Jul 15 12:11:18.084509 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.084492 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:18.099749 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.099728 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4crgq" Jul 15 12:11:18.109045 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.109028 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k2z7d" Jul 15 12:11:18.222112 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:18.222075 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c7bc3b4c133f1b0578384be47f8d38.slice/crio-e668eae76fd3786ab791d1241d9a5ceb4a0dad4b71f03f1124ac63f3b1395f7c WatchSource:0}: Error finding container e668eae76fd3786ab791d1241d9a5ceb4a0dad4b71f03f1124ac63f3b1395f7c: Status 404 returned error can't find the container with id e668eae76fd3786ab791d1241d9a5ceb4a0dad4b71f03f1124ac63f3b1395f7c Jul 15 12:11:18.225829 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.225814 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 12:11:18.246583 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.246558 2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 12:11:18.299837 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.299752 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs\") pod \"network-metrics-daemon-kfbnz\" (UID: \"e79deb5e-778c-420d-a65d-671a7d3d28db\") " pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:18.299974 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:18.299879 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:11:18.299974 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:18.299930 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs podName:e79deb5e-778c-420d-a65d-671a7d3d28db nodeName:}" failed. No retries permitted until 2026-07-15 12:11:19.299915011 +0000 UTC m=+2.934309935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs") pod "network-metrics-daemon-kfbnz" (UID: "e79deb5e-778c-420d-a65d-671a7d3d28db") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:11:18.377586 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:18.377560 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda745f4b_00f5_4b21_91f3_64907b52ebea.slice/crio-0c0559acac9e754a377fb4142f8cdc349ecffcf4fa4f270295dce7901fdddf21 WatchSource:0}: Error finding container 0c0559acac9e754a377fb4142f8cdc349ecffcf4fa4f270295dce7901fdddf21: Status 404 returned error can't find the container with id 0c0559acac9e754a377fb4142f8cdc349ecffcf4fa4f270295dce7901fdddf21 Jul 15 12:11:18.400527 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.400497 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret\") pod \"global-pull-secret-syncer-sj4hd\" (UID: \"24a77a8a-6ec5-4558-8ce8-96ae7f4c9547\") " pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:18.400652 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:18.400608 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:11:18.400707 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:18.400663 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret podName:24a77a8a-6ec5-4558-8ce8-96ae7f4c9547 nodeName:}" failed. No retries permitted until 2026-07-15 12:11:19.40064606 +0000 UTC m=+3.035040984 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret") pod "global-pull-secret-syncer-sj4hd" (UID: "24a77a8a-6ec5-4558-8ce8-96ae7f4c9547") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:11:18.421527 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:18.421497 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757d71c2_a7e0_4ca7_815a_fa1c18f4a8f8.slice/crio-ce1310a749c4637637ee594afb5c8e20855097ad48ab2342c96b34503e7b7e65 WatchSource:0}: Error finding container ce1310a749c4637637ee594afb5c8e20855097ad48ab2342c96b34503e7b7e65: Status 404 returned error can't find the container with id ce1310a749c4637637ee594afb5c8e20855097ad48ab2342c96b34503e7b7e65 Jul 15 12:11:18.501113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.501088 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqmtr\" (UniqueName: \"kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr\") pod \"network-check-target-hqq7p\" (UID: \"ad5f3fa8-e4d3-42bf-8b16-94961009814d\") " pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:18.501257 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:18.501190 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:11:18.501257 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:18.501208 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:11:18.501257 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:18.501216 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mqmtr for pod openshift-network-diagnostics/network-check-target-hqq7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:11:18.501257 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:18.501256 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr podName:ad5f3fa8-e4d3-42bf-8b16-94961009814d nodeName:}" failed. No retries permitted until 2026-07-15 12:11:19.501243976 +0000 UTC m=+3.135638900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mqmtr" (UniqueName: "kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr") pod "network-check-target-hqq7p" (UID: "ad5f3fa8-e4d3-42bf-8b16-94961009814d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:11:18.547613 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:18.547581 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f10944_d216_4dc4_8455_7460d9849743.slice/crio-a76c6a503e99d194a80ce21f94db4d6023ced9c1163f4387dc0c8ba758db724d WatchSource:0}: Error finding container a76c6a503e99d194a80ce21f94db4d6023ced9c1163f4387dc0c8ba758db724d: Status 404 returned error can't find the container with id a76c6a503e99d194a80ce21f94db4d6023ced9c1163f4387dc0c8ba758db724d Jul 15 12:11:18.586317 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:18.586294 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b43d5f526b7974ac738b5aaab0d959.slice/crio-01e5c4dfad6ce3f15cefa0b5572afcbe52b9a6b0890b64c2e85e73ff3edd0ed7 WatchSource:0}: Error finding container 01e5c4dfad6ce3f15cefa0b5572afcbe52b9a6b0890b64c2e85e73ff3edd0ed7: Status 404 returned error can't find the container with id 01e5c4dfad6ce3f15cefa0b5572afcbe52b9a6b0890b64c2e85e73ff3edd0ed7 Jul 15 12:11:18.703982 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:18.703960 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf476f12b_f4b6_4c20_b26c_2c88eac44e2d.slice/crio-8334d94e4041354e514f4a2902a7e20140447f1f4a5337873cf540cb2a78004b WatchSource:0}: Error finding container 8334d94e4041354e514f4a2902a7e20140447f1f4a5337873cf540cb2a78004b: Status 404 returned error can't find the container with id 8334d94e4041354e514f4a2902a7e20140447f1f4a5337873cf540cb2a78004b Jul 15 12:11:18.712195 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:18.712174 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda434baae_76d0_466a_b7e5_7e0e1e57d90e.slice/crio-e6610a50c5f81fe80ad813e9072d25a369da1434481356ac592de9df90299105 WatchSource:0}: Error finding container e6610a50c5f81fe80ad813e9072d25a369da1434481356ac592de9df90299105: Status 404 returned error can't find the container with id e6610a50c5f81fe80ad813e9072d25a369da1434481356ac592de9df90299105 Jul 15 12:11:18.726234 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.726206 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 12:06:17 +0000 UTC" deadline="2027-12-13 23:15:37.559852578 +0000 UTC" Jul 15 12:11:18.726318 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.726236 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12395h4m18.8336204s" Jul 15 12:11:18.848557 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.848482 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:18.848557 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.848496 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:18.848739 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:18.848577 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqq7p" podUID="ad5f3fa8-e4d3-42bf-8b16-94961009814d" Jul 15 12:11:18.848739 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:18.848691 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfbnz" podUID="e79deb5e-778c-420d-a65d-671a7d3d28db" Jul 15 12:11:18.852181 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.852145 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2z7d" event={"ID":"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8","Type":"ContainerStarted","Data":"ce1310a749c4637637ee594afb5c8e20855097ad48ab2342c96b34503e7b7e65"} Jul 15 12:11:18.855469 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.855446 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4zpzj" event={"ID":"da745f4b-00f5-4b21-91f3-64907b52ebea","Type":"ContainerStarted","Data":"0c0559acac9e754a377fb4142f8cdc349ecffcf4fa4f270295dce7901fdddf21"} Jul 15 12:11:18.856464 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.856443 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" event={"ID":"a1c7bc3b4c133f1b0578384be47f8d38","Type":"ContainerStarted","Data":"e668eae76fd3786ab791d1241d9a5ceb4a0dad4b71f03f1124ac63f3b1395f7c"} Jul 15 12:11:18.857467 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.857449 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" event={"ID":"a434baae-76d0-466a-b7e5-7e0e1e57d90e","Type":"ContainerStarted","Data":"e6610a50c5f81fe80ad813e9072d25a369da1434481356ac592de9df90299105"} Jul 15 12:11:18.858357 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.858342 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4crgq" event={"ID":"f476f12b-f4b6-4c20-b26c-2c88eac44e2d","Type":"ContainerStarted","Data":"8334d94e4041354e514f4a2902a7e20140447f1f4a5337873cf540cb2a78004b"} Jul 15 12:11:18.859181 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.859166 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" event={"ID":"34b43d5f526b7974ac738b5aaab0d959","Type":"ContainerStarted","Data":"01e5c4dfad6ce3f15cefa0b5572afcbe52b9a6b0890b64c2e85e73ff3edd0ed7"} Jul 15 12:11:18.860037 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:18.860015 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" event={"ID":"44f10944-d216-4dc4-8455-7460d9849743","Type":"ContainerStarted","Data":"a76c6a503e99d194a80ce21f94db4d6023ced9c1163f4387dc0c8ba758db724d"} Jul 15 12:11:19.049192 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:19.049165 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 12:11:19.195932 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:19.195901 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15dd1aa_a5bc_4f58_9444_8b1f69f80f5e.slice/crio-51199fa32dfbaf7e88eefe1d32980d648603ac6acc41e4ac6d6a083cd17e0fb7 WatchSource:0}: Error finding container 51199fa32dfbaf7e88eefe1d32980d648603ac6acc41e4ac6d6a083cd17e0fb7: Status 404 returned error can't find the container with id 51199fa32dfbaf7e88eefe1d32980d648603ac6acc41e4ac6d6a083cd17e0fb7 Jul 15 12:11:19.272382 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:19.271647 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27881d83_289d_4a88_beba_585f786c0e0c.slice/crio-b4a0e39d98d4abd7ce5222cd28ea5b0d98bbf99daa65ea65685391380ef498ff WatchSource:0}: Error finding container b4a0e39d98d4abd7ce5222cd28ea5b0d98bbf99daa65ea65685391380ef498ff: Status 404 returned error can't find the container with id b4a0e39d98d4abd7ce5222cd28ea5b0d98bbf99daa65ea65685391380ef498ff Jul 15 12:11:19.308470 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:19.308441 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs\") pod \"network-metrics-daemon-kfbnz\" (UID: \"e79deb5e-778c-420d-a65d-671a7d3d28db\") " pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:19.308614 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:19.308555 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:11:19.308614 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:19.308602 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs podName:e79deb5e-778c-420d-a65d-671a7d3d28db nodeName:}" failed. No retries permitted until 2026-07-15 12:11:21.30858935 +0000 UTC m=+4.942984274 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs") pod "network-metrics-daemon-kfbnz" (UID: "e79deb5e-778c-420d-a65d-671a7d3d28db") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:11:19.387024 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:19.386993 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6132cce3_5328_44d3_af00_a59e66e03f62.slice/crio-f5eb2ee5146f8fb88cb3fa0fd952208ed0233ca636ebbc99901d1f277ff10596 WatchSource:0}: Error finding container f5eb2ee5146f8fb88cb3fa0fd952208ed0233ca636ebbc99901d1f277ff10596: Status 404 returned error can't find the container with id f5eb2ee5146f8fb88cb3fa0fd952208ed0233ca636ebbc99901d1f277ff10596 Jul 15 12:11:19.414715 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:19.409716 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret\") pod \"global-pull-secret-syncer-sj4hd\" (UID: \"24a77a8a-6ec5-4558-8ce8-96ae7f4c9547\") " pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:19.414715 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:19.409847 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:11:19.414715 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:19.409891 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret podName:24a77a8a-6ec5-4558-8ce8-96ae7f4c9547 nodeName:}" failed. No retries permitted until 2026-07-15 12:11:21.409878059 +0000 UTC m=+5.044272983 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret") pod "global-pull-secret-syncer-sj4hd" (UID: "24a77a8a-6ec5-4558-8ce8-96ae7f4c9547") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:11:19.510564 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:19.510482 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqmtr\" (UniqueName: \"kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr\") pod \"network-check-target-hqq7p\" (UID: \"ad5f3fa8-e4d3-42bf-8b16-94961009814d\") " pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:19.510714 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:19.510667 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:11:19.510714 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:19.510685 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:11:19.510714 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:19.510696 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mqmtr for pod openshift-network-diagnostics/network-check-target-hqq7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:11:19.510885 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:19.510758 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr podName:ad5f3fa8-e4d3-42bf-8b16-94961009814d nodeName:}" failed. No retries permitted until 2026-07-15 12:11:21.510741297 +0000 UTC m=+5.145136233 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mqmtr" (UniqueName: "kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr") pod "network-check-target-hqq7p" (UID: "ad5f3fa8-e4d3-42bf-8b16-94961009814d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:11:19.726899 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:19.726827 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 12:06:17 +0000 UTC" deadline="2028-04-25 20:35:35.717936726 +0000 UTC" Jul 15 12:11:19.726899 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:19.726888 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15608h24m15.991052551s" Jul 15 12:11:19.849349 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:19.849273 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:19.849497 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:19.849404 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4hd" podUID="24a77a8a-6ec5-4558-8ce8-96ae7f4c9547" Jul 15 12:11:19.875513 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:19.875483 2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 12:11:19.883112 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:19.883052 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8ngfj" event={"ID":"6132cce3-5328-44d3-af00-a59e66e03f62","Type":"ContainerStarted","Data":"f5eb2ee5146f8fb88cb3fa0fd952208ed0233ca636ebbc99901d1f277ff10596"} Jul 15 12:11:19.893515 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:19.893427 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p8nqm" event={"ID":"6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8","Type":"ContainerStarted","Data":"acc0e2bf27ab812e96bf224138eb2d8b66625801657354fcd67c538bb3fed505"} Jul 15 12:11:19.895743 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:19.895718 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b8nd7" event={"ID":"a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e","Type":"ContainerStarted","Data":"51199fa32dfbaf7e88eefe1d32980d648603ac6acc41e4ac6d6a083cd17e0fb7"} Jul 15 12:11:19.897912 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:19.897888 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" event={"ID":"27881d83-289d-4a88-beba-585f786c0e0c","Type":"ContainerStarted","Data":"b4a0e39d98d4abd7ce5222cd28ea5b0d98bbf99daa65ea65685391380ef498ff"} Jul 15 12:11:20.792620 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:20.792298 2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 12:11:20.849904 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:20.849766 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:20.850063 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:20.850027 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfbnz" podUID="e79deb5e-778c-420d-a65d-671a7d3d28db" Jul 15 12:11:20.850678 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:20.850455 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:20.850678 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:20.850548 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqq7p" podUID="ad5f3fa8-e4d3-42bf-8b16-94961009814d" Jul 15 12:11:21.327791 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:21.327746 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs\") pod \"network-metrics-daemon-kfbnz\" (UID: \"e79deb5e-778c-420d-a65d-671a7d3d28db\") " pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:21.328046 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:21.327999 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:11:21.328134 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:21.328064 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs podName:e79deb5e-778c-420d-a65d-671a7d3d28db nodeName:}" failed. No retries permitted until 2026-07-15 12:11:25.328043334 +0000 UTC m=+8.962438308 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs") pod "network-metrics-daemon-kfbnz" (UID: "e79deb5e-778c-420d-a65d-671a7d3d28db") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:11:21.428849 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:21.428810 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret\") pod \"global-pull-secret-syncer-sj4hd\" (UID: \"24a77a8a-6ec5-4558-8ce8-96ae7f4c9547\") " pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:21.429022 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:21.428987 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:11:21.429087 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:21.429052 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret podName:24a77a8a-6ec5-4558-8ce8-96ae7f4c9547 nodeName:}" failed. No retries permitted until 2026-07-15 12:11:25.429035665 +0000 UTC m=+9.063430608 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret") pod "global-pull-secret-syncer-sj4hd" (UID: "24a77a8a-6ec5-4558-8ce8-96ae7f4c9547") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:11:21.529637 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:21.529606 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqmtr\" (UniqueName: \"kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr\") pod \"network-check-target-hqq7p\" (UID: \"ad5f3fa8-e4d3-42bf-8b16-94961009814d\") " pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:21.529805 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:21.529731 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:11:21.529805 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:21.529746 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:11:21.529805 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:21.529754 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mqmtr for pod openshift-network-diagnostics/network-check-target-hqq7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:11:21.529957 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:21.529822 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr podName:ad5f3fa8-e4d3-42bf-8b16-94961009814d nodeName:}" failed. No retries permitted until 2026-07-15 12:11:25.529804991 +0000 UTC m=+9.164199919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mqmtr" (UniqueName: "kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr") pod "network-check-target-hqq7p" (UID: "ad5f3fa8-e4d3-42bf-8b16-94961009814d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:11:21.848555 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:21.848524 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:21.848989 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:21.848663 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4hd" podUID="24a77a8a-6ec5-4558-8ce8-96ae7f4c9547" Jul 15 12:11:22.849553 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:22.848889 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:22.849553 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:22.849034 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfbnz" podUID="e79deb5e-778c-420d-a65d-671a7d3d28db" Jul 15 12:11:22.849553 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:22.849483 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:22.854342 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:22.850708 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqq7p" podUID="ad5f3fa8-e4d3-42bf-8b16-94961009814d" Jul 15 12:11:23.848291 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:23.848255 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:23.848479 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:23.848388 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4hd" podUID="24a77a8a-6ec5-4558-8ce8-96ae7f4c9547" Jul 15 12:11:24.849947 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:24.849911 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:24.850339 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:24.850057 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfbnz" podUID="e79deb5e-778c-420d-a65d-671a7d3d28db" Jul 15 12:11:24.850339 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:24.850222 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:24.850339 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:24.850308 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqq7p" podUID="ad5f3fa8-e4d3-42bf-8b16-94961009814d" Jul 15 12:11:25.362331 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:25.362247 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs\") pod \"network-metrics-daemon-kfbnz\" (UID: \"e79deb5e-778c-420d-a65d-671a7d3d28db\") " pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:25.362472 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:25.362430 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:11:25.362522 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:25.362493 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs podName:e79deb5e-778c-420d-a65d-671a7d3d28db nodeName:}" failed. No retries permitted until 2026-07-15 12:11:33.36247382 +0000 UTC m=+16.996868753 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs") pod "network-metrics-daemon-kfbnz" (UID: "e79deb5e-778c-420d-a65d-671a7d3d28db") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:11:25.463100 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:25.463059 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret\") pod \"global-pull-secret-syncer-sj4hd\" (UID: \"24a77a8a-6ec5-4558-8ce8-96ae7f4c9547\") " pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:25.463308 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:25.463203 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:11:25.463308 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:25.463268 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret podName:24a77a8a-6ec5-4558-8ce8-96ae7f4c9547 nodeName:}" failed. No retries permitted until 2026-07-15 12:11:33.463251975 +0000 UTC m=+17.097646903 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret") pod "global-pull-secret-syncer-sj4hd" (UID: "24a77a8a-6ec5-4558-8ce8-96ae7f4c9547") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:11:25.563872 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:25.563833 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqmtr\" (UniqueName: \"kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr\") pod \"network-check-target-hqq7p\" (UID: \"ad5f3fa8-e4d3-42bf-8b16-94961009814d\") " pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:25.564059 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:25.564004 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:11:25.564059 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:25.564027 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:11:25.564059 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:25.564039 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mqmtr for pod openshift-network-diagnostics/network-check-target-hqq7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:11:25.564234 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:25.564106 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr podName:ad5f3fa8-e4d3-42bf-8b16-94961009814d nodeName:}" failed. No retries permitted until 2026-07-15 12:11:33.564086914 +0000 UTC m=+17.198481860 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mqmtr" (UniqueName: "kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr") pod "network-check-target-hqq7p" (UID: "ad5f3fa8-e4d3-42bf-8b16-94961009814d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:11:25.848476 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:25.848409 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:25.848628 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:25.848524 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4hd" podUID="24a77a8a-6ec5-4558-8ce8-96ae7f4c9547" Jul 15 12:11:26.851416 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:26.850861 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:26.851416 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:26.851003 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfbnz" podUID="e79deb5e-778c-420d-a65d-671a7d3d28db" Jul 15 12:11:26.852077 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:26.851939 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:26.852077 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:26.852042 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqq7p" podUID="ad5f3fa8-e4d3-42bf-8b16-94961009814d" Jul 15 12:11:27.848610 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:27.848577 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:27.848886 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:27.848703 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4hd" podUID="24a77a8a-6ec5-4558-8ce8-96ae7f4c9547" Jul 15 12:11:28.849159 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:28.849120 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:28.849583 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:28.849268 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfbnz" podUID="e79deb5e-778c-420d-a65d-671a7d3d28db" Jul 15 12:11:28.849711 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:28.849692 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:28.849830 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:28.849808 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqq7p" podUID="ad5f3fa8-e4d3-42bf-8b16-94961009814d" Jul 15 12:11:29.848497 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:29.848460 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:29.848694 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:29.848604 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4hd" podUID="24a77a8a-6ec5-4558-8ce8-96ae7f4c9547" Jul 15 12:11:30.849156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:30.849117 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:30.849614 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:30.849117 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:30.849614 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:30.849283 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfbnz" podUID="e79deb5e-778c-420d-a65d-671a7d3d28db" Jul 15 12:11:30.849614 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:30.849337 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqq7p" podUID="ad5f3fa8-e4d3-42bf-8b16-94961009814d" Jul 15 12:11:31.849157 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:31.849124 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:31.849318 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:31.849249 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4hd" podUID="24a77a8a-6ec5-4558-8ce8-96ae7f4c9547" Jul 15 12:11:32.848638 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:32.848607 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:32.848824 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:32.848745 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfbnz" podUID="e79deb5e-778c-420d-a65d-671a7d3d28db" Jul 15 12:11:32.848824 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:32.848802 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:32.848938 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:32.848900 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqq7p" podUID="ad5f3fa8-e4d3-42bf-8b16-94961009814d" Jul 15 12:11:33.421918 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:33.421870 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs\") pod \"network-metrics-daemon-kfbnz\" (UID: \"e79deb5e-778c-420d-a65d-671a7d3d28db\") " pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:33.422326 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:33.422014 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:11:33.422326 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:33.422084 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs podName:e79deb5e-778c-420d-a65d-671a7d3d28db nodeName:}" failed. No retries permitted until 2026-07-15 12:11:49.422064508 +0000 UTC m=+33.056459435 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs") pod "network-metrics-daemon-kfbnz" (UID: "e79deb5e-778c-420d-a65d-671a7d3d28db") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:11:33.522656 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:33.522611 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret\") pod \"global-pull-secret-syncer-sj4hd\" (UID: \"24a77a8a-6ec5-4558-8ce8-96ae7f4c9547\") " pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:33.522851 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:33.522738 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:11:33.522851 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:33.522836 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret podName:24a77a8a-6ec5-4558-8ce8-96ae7f4c9547 nodeName:}" failed. No retries permitted until 2026-07-15 12:11:49.52281531 +0000 UTC m=+33.157210245 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret") pod "global-pull-secret-syncer-sj4hd" (UID: "24a77a8a-6ec5-4558-8ce8-96ae7f4c9547") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:11:33.623852 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:33.623816 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqmtr\" (UniqueName: \"kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr\") pod \"network-check-target-hqq7p\" (UID: \"ad5f3fa8-e4d3-42bf-8b16-94961009814d\") " pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:33.624065 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:33.624040 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:11:33.624065 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:33.624068 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:11:33.624249 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:33.624082 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mqmtr for pod openshift-network-diagnostics/network-check-target-hqq7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:11:33.624249 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:33.624143 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr podName:ad5f3fa8-e4d3-42bf-8b16-94961009814d nodeName:}" failed. No retries permitted until 2026-07-15 12:11:49.62412561 +0000 UTC m=+33.258520555 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mqmtr" (UniqueName: "kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr") pod "network-check-target-hqq7p" (UID: "ad5f3fa8-e4d3-42bf-8b16-94961009814d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:11:33.848949 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:33.848875 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:33.849094 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:33.848997 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4hd" podUID="24a77a8a-6ec5-4558-8ce8-96ae7f4c9547" Jul 15 12:11:34.848710 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:34.848671 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:34.849140 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:34.848727 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:34.849140 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:34.848837 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfbnz" podUID="e79deb5e-778c-420d-a65d-671a7d3d28db" Jul 15 12:11:34.849140 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:34.848964 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqq7p" podUID="ad5f3fa8-e4d3-42bf-8b16-94961009814d" Jul 15 12:11:35.848558 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:35.848524 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:35.848738 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:35.848692 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4hd" podUID="24a77a8a-6ec5-4558-8ce8-96ae7f4c9547" Jul 15 12:11:36.849222 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:36.849184 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:36.849681 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:36.849309 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfbnz" podUID="e79deb5e-778c-420d-a65d-671a7d3d28db" Jul 15 12:11:36.849681 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:36.849371 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:36.849681 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:36.849475 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqq7p" podUID="ad5f3fa8-e4d3-42bf-8b16-94961009814d" Jul 15 12:11:37.849263 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:37.848845 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:37.849263 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:37.848976 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4hd" podUID="24a77a8a-6ec5-4558-8ce8-96ae7f4c9547" Jul 15 12:11:38.849042 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:38.849003 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:38.849325 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:38.849011 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:38.849325 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:38.849122 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfbnz" podUID="e79deb5e-778c-420d-a65d-671a7d3d28db" Jul 15 12:11:38.849325 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:38.849236 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqq7p" podUID="ad5f3fa8-e4d3-42bf-8b16-94961009814d" Jul 15 12:11:39.848637 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:39.848354 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:39.848820 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:39.848709 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4hd" podUID="24a77a8a-6ec5-4558-8ce8-96ae7f4c9547" Jul 15 12:11:39.949223 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:39.949199 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:11:39.949878 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:39.949561 2556 generic.go:356] "Generic (PLEG): container finished" podID="a434baae-76d0-466a-b7e5-7e0e1e57d90e" containerID="3968c50559f99c7e9f5a1043f66780e1134d01ccc7e9739830e772700cdf0181" exitCode=1 Jul 15 12:11:39.949878 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:39.949627 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" event={"ID":"a434baae-76d0-466a-b7e5-7e0e1e57d90e","Type":"ContainerStarted","Data":"31464ff43c0ea40eb256c85ad90d7416627174a893876f0c5bb145589b4dc607"} Jul 15 12:11:39.949878 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:39.949694 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" event={"ID":"a434baae-76d0-466a-b7e5-7e0e1e57d90e","Type":"ContainerStarted","Data":"b6088c3ae9c4ea52f75c51e7386b71a92ceaae9a188626329ef4a18e18191a8c"} Jul 15 12:11:39.949878 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:39.949735 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" event={"ID":"a434baae-76d0-466a-b7e5-7e0e1e57d90e","Type":"ContainerStarted","Data":"05e54b689f2b0998eae6584390bc1a1b2ce4ae0cdb8f8b73adb94c13d69dfec4"} Jul 15 12:11:39.949878 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:39.949791 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" event={"ID":"a434baae-76d0-466a-b7e5-7e0e1e57d90e","Type":"ContainerStarted","Data":"bc096c5b0cddbbd9b6ce704eb7540d62f4b63b6af911dbe96add4df83b4ac689"} Jul 15 12:11:39.949878 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:39.949826 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" event={"ID":"a434baae-76d0-466a-b7e5-7e0e1e57d90e","Type":"ContainerDied","Data":"3968c50559f99c7e9f5a1043f66780e1134d01ccc7e9739830e772700cdf0181"} Jul 15 12:11:39.949878 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:39.949870 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" event={"ID":"a434baae-76d0-466a-b7e5-7e0e1e57d90e","Type":"ContainerStarted","Data":"9a78909ee572625d3ded8d5e136fa1e3b0419891dc69bd99e83bdd1648f0b4c9"} Jul 15 12:11:39.950983 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:39.950962 2556 generic.go:356] "Generic (PLEG): container finished" podID="34b43d5f526b7974ac738b5aaab0d959" containerID="a41b4751ca0f47008150152913e1678342c86c59bb953562b475b7d077748a77" exitCode=0 Jul 15 12:11:39.951068 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:39.951031 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" event={"ID":"34b43d5f526b7974ac738b5aaab0d959","Type":"ContainerDied","Data":"a41b4751ca0f47008150152913e1678342c86c59bb953562b475b7d077748a77"} Jul 15 12:11:39.952428 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:39.952398 2556 generic.go:356] "Generic (PLEG): container finished" podID="757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8" containerID="6d14d04a8758f413d1996e7ec954c6ac970942c154f41c5b1f0d601fdf438aa6" exitCode=0 Jul 15 12:11:39.952528 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:39.952474 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2z7d" event={"ID":"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8","Type":"ContainerDied","Data":"6d14d04a8758f413d1996e7ec954c6ac970942c154f41c5b1f0d601fdf438aa6"} Jul 15 12:11:39.954384 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:39.953937 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8ngfj" event={"ID":"6132cce3-5328-44d3-af00-a59e66e03f62","Type":"ContainerStarted","Data":"8f53143eef016f3149e95998d1f38419b4469f93d279420429285cb8d84cc14c"} Jul 15 12:11:39.955261 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:39.955232 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p8nqm" event={"ID":"6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8","Type":"ContainerStarted","Data":"a119b4f79ac8a2ec52aaa1c2ee796e1753ac80572689c799987c6fc01e50df2b"} Jul 15 12:11:39.956940 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:39.956918 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b8nd7" event={"ID":"a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e","Type":"ContainerStarted","Data":"c851dde38284fe242a3f9cd7ed522304e4d7fe661ac8b4d76403864e227b72d7"} Jul 15 12:11:39.958486 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:39.958468 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" event={"ID":"44f10944-d216-4dc4-8455-7460d9849743","Type":"ContainerStarted","Data":"ee141091a7420a4d77377673a32d83bd80e272f006e9ff29991a5c352c13edd7"} Jul 15 12:11:39.959886 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:39.959801 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4zpzj" event={"ID":"da745f4b-00f5-4b21-91f3-64907b52ebea","Type":"ContainerStarted","Data":"7e1f7dd536faaa8b89529703f92f41cde9b8bea705946472d4c30b06f11e6aef"} Jul 15 12:11:39.961251 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:39.961223 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" event={"ID":"a1c7bc3b4c133f1b0578384be47f8d38","Type":"ContainerStarted","Data":"66f1be8c6b08b8eef5ed23c2a14dce1078169b2c50ffea55e410da14aeabff2d"} Jul 15 12:11:39.962684 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:39.962666 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" event={"ID":"27881d83-289d-4a88-beba-585f786c0e0c","Type":"ContainerStarted","Data":"4d30f20418de414bf3dfa9e0cfcb2c54409de3dd1841c764233ab1fe4f701c65"} Jul 15 12:11:40.005346 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:40.004474 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8ngfj" podStartSLOduration=4.224908755 podStartE2EDuration="24.004453813s" totalImagesPullingTime="19.779545058s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:11:39.990368289 +0000 UTC m=+23.624763237" watchObservedRunningTime="2026-07-15 12:11:40.004453813 +0000 UTC m=+23.638848765" Jul 15 12:11:40.005346 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:40.005077 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4zpzj" podStartSLOduration=12.003177462 podStartE2EDuration="24.005063885s" totalImagesPullingTime="12.001886423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:11:40.004433624 +0000 UTC m=+23.638828573" watchObservedRunningTime="2026-07-15 12:11:40.005063885 +0000 UTC m=+23.639458834" Jul 15 12:11:40.019589 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:40.019541 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-p8nqm" podStartSLOduration=4.371276134 podStartE2EDuration="24.01952775s" totalImagesPullingTime="19.648251616s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:11:40.019024317 +0000 UTC m=+23.653419267" watchObservedRunningTime="2026-07-15 12:11:40.01952775 +0000 UTC m=+23.653922698" Jul 15 12:11:40.034171 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:40.034128 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-b8nd7" podStartSLOduration=4.23193832 podStartE2EDuration="24.034115378s" totalImagesPullingTime="19.802177058s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:11:40.03317369 +0000 UTC m=+23.667568642" watchObservedRunningTime="2026-07-15 12:11:40.034115378 +0000 UTC m=+23.668510326" Jul 15 12:11:40.051706 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:40.051664 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" podStartSLOduration=23.051647698 podStartE2EDuration="23.051647698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:11:40.051407084 +0000 UTC m=+23.685802033" watchObservedRunningTime="2026-07-15 12:11:40.051647698 +0000 UTC m=+23.686042647" Jul 15 12:11:40.069152 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:40.069106 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ndgsc" podStartSLOduration=4.322939886 podStartE2EDuration="24.069087823s" totalImagesPullingTime="19.746147937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:11:40.069066849 +0000 UTC m=+23.703461798" watchObservedRunningTime="2026-07-15 12:11:40.069087823 +0000 UTC m=+23.703482772" Jul 15 12:11:40.438236 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:40.438208 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:40.489247 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:40.489220 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:40.837593 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:40.837571 2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 12:11:40.849311 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:40.849287 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:40.849429 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:40.849407 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfbnz" podUID="e79deb5e-778c-420d-a65d-671a7d3d28db" Jul 15 12:11:40.849491 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:40.849439 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:40.849542 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:40.849497 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqq7p" podUID="ad5f3fa8-e4d3-42bf-8b16-94961009814d" Jul 15 12:11:40.967015 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:40.966933 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4crgq" event={"ID":"f476f12b-f4b6-4c20-b26c-2c88eac44e2d","Type":"ContainerStarted","Data":"fe56ae195fad85509c9a0f542c86b5ab6100554aad687b6843d54b45a6a8f092"} Jul 15 12:11:40.968986 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:40.968952 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" event={"ID":"44f10944-d216-4dc4-8455-7460d9849743","Type":"ContainerStarted","Data":"38a6a6e48c5ef17e03fbeb64f8145621d8028067287d5cc31133f6f19e7c05ac"} Jul 15 12:11:40.997668 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:40.997617 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4crgq" podStartSLOduration=8.738213167 podStartE2EDuration="24.997597026s" totalImagesPullingTime="16.259383859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:11:40.982075981 +0000 UTC m=+24.616470928" watchObservedRunningTime="2026-07-15 12:11:40.997597026 +0000 UTC m=+24.631991975" Jul 15 12:11:41.201514 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:41.201481 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:41.405869 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:41.405840 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:41.766057 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:41.765972 2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T12:11:40.837590022Z","UUID":"01daf7a2-d085-460f-8a7a-e00db84de0e9","Handler":null,"Name":"","Endpoint":""} Jul 15 12:11:41.767620 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:41.767598 2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 12:11:41.767767 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:41.767641 2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 12:11:41.849153 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:41.849111 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:41.849312 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:41.849233 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4hd" podUID="24a77a8a-6ec5-4558-8ce8-96ae7f4c9547" Jul 15 12:11:41.860626 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:41.860590 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:41.975429 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:41.975397 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" event={"ID":"34b43d5f526b7974ac738b5aaab0d959","Type":"ContainerStarted","Data":"afcb0261527555ef054a38c40dc71fbe68e3dc1879dcf8b34bf682ef84c3109d"} Jul 15 12:11:42.849078 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:42.848887 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:42.849392 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:42.848887 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:42.849392 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:42.849177 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqq7p" podUID="ad5f3fa8-e4d3-42bf-8b16-94961009814d" Jul 15 12:11:42.849392 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:42.849239 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfbnz" podUID="e79deb5e-778c-420d-a65d-671a7d3d28db" Jul 15 12:11:42.979590 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:42.979547 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" event={"ID":"44f10944-d216-4dc4-8455-7460d9849743","Type":"ContainerStarted","Data":"1e08ca2af9f4839e396030ad90d69322cfc7dd8c6b4452b75ea0214950926b92"} Jul 15 12:11:42.983272 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:42.983252 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:11:42.983648 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:42.983618 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" event={"ID":"a434baae-76d0-466a-b7e5-7e0e1e57d90e","Type":"ContainerStarted","Data":"d46f3353c6912a9ea5114894659f088c3752e3e12e8c0b9df8eee4aa4b68f1ef"} Jul 15 12:11:42.998064 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:42.998025 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" podStartSLOduration=25.998010378 podStartE2EDuration="25.998010378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:11:41.99198972 +0000 UTC m=+25.626384668" watchObservedRunningTime="2026-07-15 12:11:42.998010378 +0000 UTC m=+26.632405326" Jul 15 12:11:42.998577 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:42.998537 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jw5x" podStartSLOduration=3.979551102 podStartE2EDuration="26.998529502s" totalImagesPullingTime="23.0189784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:11:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:11:42.997873341 +0000 UTC m=+26.632268293" watchObservedRunningTime="2026-07-15 12:11:42.998529502 +0000 UTC m=+26.632924498" Jul 15 12:11:43.716933 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:43.716897 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:43.779224 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:43.779191 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:43.848911 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:43.848882 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:43.849093 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:43.848996 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4hd" podUID="24a77a8a-6ec5-4558-8ce8-96ae7f4c9547" Jul 15 12:11:43.991227 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:43.991157 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4zpzj" Jul 15 12:11:43.992019 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:43.992000 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4zpzj" Jul 15 12:11:44.848891 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:44.848722 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:44.849058 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:44.848750 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:44.849058 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:44.848972 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfbnz" podUID="e79deb5e-778c-420d-a65d-671a7d3d28db" Jul 15 12:11:44.849058 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:44.849037 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqq7p" podUID="ad5f3fa8-e4d3-42bf-8b16-94961009814d" Jul 15 12:11:44.993179 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:44.993116 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:11:44.993887 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:44.993467 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" event={"ID":"a434baae-76d0-466a-b7e5-7e0e1e57d90e","Type":"ContainerStarted","Data":"abb6293979f049c61bf676fd2bc82ac83b050299f1b0537a9a31f9e2db1d9e03"} Jul 15 12:11:44.993887 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:44.993682 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:44.993887 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:44.993747 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:44.993887 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:44.993808 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:44.994092 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:44.993957 2556 scope.go:117] "RemoveContainer" containerID="3968c50559f99c7e9f5a1043f66780e1134d01ccc7e9739830e772700cdf0181" Jul 15 12:11:44.995485 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:44.995461 2556 generic.go:356] "Generic (PLEG): container finished" podID="757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8" containerID="29f6537d83e6ba5efc99b1b67e53cdf27e54e7d465cd8109b4748fdb5c771ea9" exitCode=0 Jul 15 12:11:44.995611 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:44.995523 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2z7d" event={"ID":"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8","Type":"ContainerDied","Data":"29f6537d83e6ba5efc99b1b67e53cdf27e54e7d465cd8109b4748fdb5c771ea9"} Jul 15 12:11:44.995713 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:44.995693 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4zpzj" Jul 15 12:11:44.996271 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:44.996189 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4zpzj" Jul 15 12:11:45.010194 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:45.010174 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:45.012024 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:45.012007 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:11:45.093957 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:45.093932 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:45.849229 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:45.849203 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:45.849471 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:45.849344 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4hd" podUID="24a77a8a-6ec5-4558-8ce8-96ae7f4c9547" Jul 15 12:11:46.001224 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:46.001201 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:11:46.001657 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:46.001525 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" event={"ID":"a434baae-76d0-466a-b7e5-7e0e1e57d90e","Type":"ContainerStarted","Data":"d31614f5b5614250de47aeca0489390f480797d8a5a375560783d5b7803801fc"} Jul 15 12:11:46.029459 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:46.029407 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" podStartSLOduration=9.696855864 podStartE2EDuration="30.029384642s" totalImagesPullingTime="20.332528778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:11:46.026683712 +0000 UTC m=+29.661078661" watchObservedRunningTime="2026-07-15 12:11:46.029384642 +0000 UTC m=+29.663779592" Jul 15 12:11:46.491174 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:46.490964 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sj4hd"] Jul 15 12:11:46.491336 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:46.491274 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:46.491401 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:46.491372 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4hd" podUID="24a77a8a-6ec5-4558-8ce8-96ae7f4c9547" Jul 15 12:11:46.494746 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:46.494720 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hqq7p"] Jul 15 12:11:46.494892 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:46.494841 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:46.494959 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:46.494928 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqq7p" podUID="ad5f3fa8-e4d3-42bf-8b16-94961009814d" Jul 15 12:11:46.495305 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:46.495281 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kfbnz"] Jul 15 12:11:46.495400 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:46.495387 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:46.495510 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:46.495490 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfbnz" podUID="e79deb5e-778c-420d-a65d-671a7d3d28db" Jul 15 12:11:47.005212 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:47.005176 2556 generic.go:356] "Generic (PLEG): container finished" podID="757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8" containerID="913a5bc6946d2061311befdcdf178b71cf003e60e4ac229d77bd5eb825c7b2b9" exitCode=0 Jul 15 12:11:47.005591 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:47.005262 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2z7d" event={"ID":"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8","Type":"ContainerDied","Data":"913a5bc6946d2061311befdcdf178b71cf003e60e4ac229d77bd5eb825c7b2b9"} Jul 15 12:11:47.848544 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:47.848510 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:47.848721 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:47.848632 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4hd" podUID="24a77a8a-6ec5-4558-8ce8-96ae7f4c9547" Jul 15 12:11:48.848741 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:48.848708 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:48.849493 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:48.848794 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:48.849493 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:48.848882 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqq7p" podUID="ad5f3fa8-e4d3-42bf-8b16-94961009814d" Jul 15 12:11:48.849493 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:48.848953 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfbnz" podUID="e79deb5e-778c-420d-a65d-671a7d3d28db" Jul 15 12:11:49.012101 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:49.012068 2556 generic.go:356] "Generic (PLEG): container finished" podID="757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8" containerID="54dec7f71a42c3825e35ea7ade64b4ec70d17ed6764d023602d4c4c3fb35f361" exitCode=0 Jul 15 12:11:49.012101 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:49.012097 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2z7d" event={"ID":"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8","Type":"ContainerDied","Data":"54dec7f71a42c3825e35ea7ade64b4ec70d17ed6764d023602d4c4c3fb35f361"} Jul 15 12:11:49.443521 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:49.443495 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs\") pod \"network-metrics-daemon-kfbnz\" (UID: \"e79deb5e-778c-420d-a65d-671a7d3d28db\") " pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:49.443673 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:49.443606 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:11:49.443673 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:49.443658 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs podName:e79deb5e-778c-420d-a65d-671a7d3d28db nodeName:}" failed. No retries permitted until 2026-07-15 12:12:21.443644616 +0000 UTC m=+65.078039544 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs") pod "network-metrics-daemon-kfbnz" (UID: "e79deb5e-778c-420d-a65d-671a7d3d28db") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:11:49.505812 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:49.505769 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:49.544838 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:49.544806 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret\") pod \"global-pull-secret-syncer-sj4hd\" (UID: \"24a77a8a-6ec5-4558-8ce8-96ae7f4c9547\") " pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:49.544953 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:49.544923 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:11:49.545012 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:49.544985 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret podName:24a77a8a-6ec5-4558-8ce8-96ae7f4c9547 nodeName:}" failed. No retries permitted until 2026-07-15 12:12:21.544967757 +0000 UTC m=+65.179362692 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret") pod "global-pull-secret-syncer-sj4hd" (UID: "24a77a8a-6ec5-4558-8ce8-96ae7f4c9547") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:11:49.645791 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:49.645752 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqmtr\" (UniqueName: \"kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr\") pod \"network-check-target-hqq7p\" (UID: \"ad5f3fa8-e4d3-42bf-8b16-94961009814d\") " pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:49.645917 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:49.645883 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:11:49.645917 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:49.645904 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:11:49.645917 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:49.645913 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mqmtr for pod openshift-network-diagnostics/network-check-target-hqq7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:11:49.646012 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:49.645958 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr podName:ad5f3fa8-e4d3-42bf-8b16-94961009814d nodeName:}" failed. No retries permitted until 2026-07-15 12:12:21.645945015 +0000 UTC m=+65.280339943 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mqmtr" (UniqueName: "kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr") pod "network-check-target-hqq7p" (UID: "ad5f3fa8-e4d3-42bf-8b16-94961009814d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:11:49.848525 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:49.848499 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:49.848677 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:49.848594 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4hd" podUID="24a77a8a-6ec5-4558-8ce8-96ae7f4c9547" Jul 15 12:11:50.848726 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:50.848692 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:50.849316 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:50.848703 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:50.849316 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:50.848850 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfbnz" podUID="e79deb5e-778c-420d-a65d-671a7d3d28db" Jul 15 12:11:50.849316 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:50.848950 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqq7p" podUID="ad5f3fa8-e4d3-42bf-8b16-94961009814d" Jul 15 12:11:51.728480 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.728450 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeReady" Jul 15 12:11:51.728661 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.728593 2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 12:11:51.772958 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.772917 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b89tf"] Jul 15 12:11:51.775345 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.775320 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fb9dn"] Jul 15 12:11:51.775541 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.775453 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b89tf" Jul 15 12:11:51.777562 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.777536 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fb9dn" Jul 15 12:11:51.778008 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.777988 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 12:11:51.778145 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.778104 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 12:11:51.778434 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.778415 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gfjwd\"" Jul 15 12:11:51.779987 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.779962 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xk7gb\"" Jul 15 12:11:51.780072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.780048 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 12:11:51.780470 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.780246 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 12:11:51.780470 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.780259 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 12:11:51.785162 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.785139 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fb9dn"] Jul 15 12:11:51.787634 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.787610 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b89tf"] Jul 15 12:11:51.849210 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.849185 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:11:51.851743 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.851709 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 12:11:51.860874 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.860849 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:11:51.860982 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.860886 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/989c12ef-5656-4c13-be8d-84a41e5ab36b-cert\") pod \"ingress-canary-fb9dn\" (UID: \"989c12ef-5656-4c13-be8d-84a41e5ab36b\") " pod="openshift-ingress-canary/ingress-canary-fb9dn" Jul 15 12:11:51.860982 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.860906 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-config-volume\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:11:51.860982 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.860936 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8td2d\" (UniqueName: \"kubernetes.io/projected/989c12ef-5656-4c13-be8d-84a41e5ab36b-kube-api-access-8td2d\") pod \"ingress-canary-fb9dn\" (UID: \"989c12ef-5656-4c13-be8d-84a41e5ab36b\") " pod="openshift-ingress-canary/ingress-canary-fb9dn" Jul 15 12:11:51.860982 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.860959 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh8dw\" (UniqueName: \"kubernetes.io/projected/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-kube-api-access-gh8dw\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:11:51.861193 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.861003 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-tmp-dir\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:11:51.945535 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.945460 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:51.961617 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.961595 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/989c12ef-5656-4c13-be8d-84a41e5ab36b-cert\") pod \"ingress-canary-fb9dn\" (UID: \"989c12ef-5656-4c13-be8d-84a41e5ab36b\") " pod="openshift-ingress-canary/ingress-canary-fb9dn" Jul 15 12:11:51.961690 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.961623 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-config-volume\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:11:51.961690 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.961661 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8td2d\" (UniqueName: \"kubernetes.io/projected/989c12ef-5656-4c13-be8d-84a41e5ab36b-kube-api-access-8td2d\") pod \"ingress-canary-fb9dn\" (UID: \"989c12ef-5656-4c13-be8d-84a41e5ab36b\") " pod="openshift-ingress-canary/ingress-canary-fb9dn" Jul 15 12:11:51.961690 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.961684 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh8dw\" (UniqueName: \"kubernetes.io/projected/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-kube-api-access-gh8dw\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:11:51.961805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.961714 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-tmp-dir\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:11:51.961842 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.961810 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:11:51.961925 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:51.961909 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 12:11:51.961991 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:51.961981 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls podName:7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9 nodeName:}" failed. No retries permitted until 2026-07-15 12:11:52.461956224 +0000 UTC m=+36.096351155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls") pod "dns-default-b89tf" (UID: "7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9") : secret "dns-default-metrics-tls" not found Jul 15 12:11:51.962105 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.962086 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-tmp-dir\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:11:51.962254 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.962238 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-config-volume\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:11:51.966074 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.965960 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/989c12ef-5656-4c13-be8d-84a41e5ab36b-cert\") pod \"ingress-canary-fb9dn\" (UID: \"989c12ef-5656-4c13-be8d-84a41e5ab36b\") " pod="openshift-ingress-canary/ingress-canary-fb9dn" Jul 15 12:11:51.970888 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.970864 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh8dw\" (UniqueName: \"kubernetes.io/projected/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-kube-api-access-gh8dw\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:11:51.971113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:51.971092 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8td2d\" (UniqueName: \"kubernetes.io/projected/989c12ef-5656-4c13-be8d-84a41e5ab36b-kube-api-access-8td2d\") pod \"ingress-canary-fb9dn\" (UID: \"989c12ef-5656-4c13-be8d-84a41e5ab36b\") " pod="openshift-ingress-canary/ingress-canary-fb9dn" Jul 15 12:11:52.046094 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:52.046059 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:52.094234 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:52.094197 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fb9dn" Jul 15 12:11:52.245848 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:52.245823 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fb9dn"] Jul 15 12:11:52.250365 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:11:52.250326 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989c12ef_5656_4c13_be8d_84a41e5ab36b.slice/crio-0d74d513ba05991c39ba054971ce7349422dba5fcaff95311adc505bad6b9360 WatchSource:0}: Error finding container 0d74d513ba05991c39ba054971ce7349422dba5fcaff95311adc505bad6b9360: Status 404 returned error can't find the container with id 0d74d513ba05991c39ba054971ce7349422dba5fcaff95311adc505bad6b9360 Jul 15 12:11:52.465974 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:52.465934 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:11:52.466149 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:52.466103 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 12:11:52.466219 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:52.466183 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls podName:7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9 nodeName:}" failed. No retries permitted until 2026-07-15 12:11:53.466166562 +0000 UTC m=+37.100561504 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls") pod "dns-default-b89tf" (UID: "7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9") : secret "dns-default-metrics-tls" not found Jul 15 12:11:52.848724 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:52.848685 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:11:52.848724 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:52.848716 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:11:52.852446 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:52.852222 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 12:11:52.852446 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:52.852242 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 12:11:52.853694 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:52.853673 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5qgtr\"" Jul 15 12:11:52.853921 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:52.853907 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tcn8s\"" Jul 15 12:11:52.854199 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:52.854180 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 12:11:53.023838 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:53.023801 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fb9dn" event={"ID":"989c12ef-5656-4c13-be8d-84a41e5ab36b","Type":"ContainerStarted","Data":"0d74d513ba05991c39ba054971ce7349422dba5fcaff95311adc505bad6b9360"} Jul 15 12:11:53.473761 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:53.473624 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:11:53.473963 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:53.473823 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 12:11:53.473963 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:53.473906 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls podName:7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9 nodeName:}" failed. No retries permitted until 2026-07-15 12:11:55.473881586 +0000 UTC m=+39.108276515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls") pod "dns-default-b89tf" (UID: "7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9") : secret "dns-default-metrics-tls" not found Jul 15 12:11:53.523205 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:53.523171 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:54.765679 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:54.765647 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:54.818941 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:54.818916 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:55.489389 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:55.489211 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:11:55.489389 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:55.489375 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 12:11:55.489571 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:55.489441 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls podName:7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9 nodeName:}" failed. No retries permitted until 2026-07-15 12:11:59.489425786 +0000 UTC m=+43.123820713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls") pod "dns-default-b89tf" (UID: "7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9") : secret "dns-default-metrics-tls" not found Jul 15 12:11:55.960684 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:55.960647 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:56.018647 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:56.018615 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:56.033695 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:56.033672 2556 generic.go:356] "Generic (PLEG): container finished" podID="757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8" containerID="fd240b6290e1e9b6f6ebf58729bb9cf6eb8d9e3f81cb9194a6a4983be72f01cc" exitCode=0 Jul 15 12:11:56.033839 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:56.033745 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2z7d" event={"ID":"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8","Type":"ContainerDied","Data":"fd240b6290e1e9b6f6ebf58729bb9cf6eb8d9e3f81cb9194a6a4983be72f01cc"} Jul 15 12:11:56.035380 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:56.035353 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fb9dn" event={"ID":"989c12ef-5656-4c13-be8d-84a41e5ab36b","Type":"ContainerStarted","Data":"ba49861de5a6f75157ec8fd095fff8be9f9aef395b25a74830b34d6833cff8fe"} Jul 15 12:11:56.072125 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:56.072075 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fb9dn" podStartSLOduration=2.194461427 podStartE2EDuration="5.072055301s" totalImagesPullingTime="2.877593874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:11:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:11:56.071361346 +0000 UTC m=+39.705756296" watchObservedRunningTime="2026-07-15 12:11:56.072055301 +0000 UTC m=+39.706450250" Jul 15 12:11:57.040268 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:57.040234 2556 generic.go:356] "Generic (PLEG): container finished" podID="757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8" containerID="d538f9fda14c5f7eef10b3cf7bf20dc61754fd0d122c01e856f09f729d462004" exitCode=0 Jul 15 12:11:57.040837 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:57.040319 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2z7d" event={"ID":"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8","Type":"ContainerDied","Data":"d538f9fda14c5f7eef10b3cf7bf20dc61754fd0d122c01e856f09f729d462004"} Jul 15 12:11:57.811911 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:57.811873 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:58.045809 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:58.045758 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2z7d" event={"ID":"757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8","Type":"ContainerStarted","Data":"bcf9942d5c0b00c6a40fa32e30339b64a04850351b8761cd51916f9092a0e24f"} Jul 15 12:11:58.068868 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:58.068769 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k2z7d" podStartSLOduration=8.490925605 podStartE2EDuration="42.068754462s" totalImagesPullingTime="33.577828857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:11:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:11:58.066504178 +0000 UTC m=+41.700899125" watchObservedRunningTime="2026-07-15 12:11:58.068754462 +0000 UTC m=+41.703149410" Jul 15 12:11:58.188268 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:58.188238 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:58.289704 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:58.289667 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:59.520335 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:11:59.520294 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:11:59.520690 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:59.520457 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 12:11:59.520690 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:11:59.520521 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls podName:7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9 nodeName:}" failed. No retries permitted until 2026-07-15 12:12:07.520505513 +0000 UTC m=+51.154900438 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls") pod "dns-default-b89tf" (UID: "7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9") : secret "dns-default-metrics-tls" not found Jul 15 12:12:07.577094 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:07.577050 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:12:07.577647 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:12:07.577175 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 12:12:07.577647 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:12:07.577247 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls podName:7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9 nodeName:}" failed. No retries permitted until 2026-07-15 12:12:23.57722637 +0000 UTC m=+67.211621299 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls") pod "dns-default-b89tf" (UID: "7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9") : secret "dns-default-metrics-tls" not found Jul 15 12:12:13.930014 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:13.929972 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:12:13.981217 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:13.981193 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:12:17.019072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:17.019043 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fdr4" Jul 15 12:12:17.281199 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:17.281095 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:12:17.330087 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:17.330063 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:12:18.383698 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.383655 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr"] Jul 15 12:12:18.386967 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.386937 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" Jul 15 12:12:18.390024 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.389997 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 15 12:12:18.390230 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.390216 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 15 12:12:18.390301 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.390277 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 15 12:12:18.390349 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.390277 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 15 12:12:18.395422 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.395394 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr"] Jul 15 12:12:18.410241 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.410218 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b8779674-g29jj"] Jul 15 12:12:18.411758 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.411743 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45"] Jul 15 12:12:18.411887 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.411872 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b8779674-g29jj" Jul 15 12:12:18.413513 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.413497 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:12:18.414836 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.414822 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 15 12:12:18.414918 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.414862 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-9tqpr\"" Jul 15 12:12:18.415989 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.415970 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 15 12:12:18.416088 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.415993 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 15 12:12:18.416271 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.416257 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 15 12:12:18.416344 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.416285 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 15 12:12:18.423740 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.423721 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b8779674-g29jj"] Jul 15 12:12:18.424412 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.424391 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45"] Jul 15 12:12:18.556256 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.556221 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c822d778-a0fc-4d65-8ba4-d78dc9a11ca2-ca\") pod \"cluster-proxy-proxy-agent-c74dddb88-5dq45\" (UID: \"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:12:18.556256 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.556255 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c822d778-a0fc-4d65-8ba4-d78dc9a11ca2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c74dddb88-5dq45\" (UID: \"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:12:18.556448 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.556288 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr6x4\" (UniqueName: \"kubernetes.io/projected/c822d778-a0fc-4d65-8ba4-d78dc9a11ca2-kube-api-access-mr6x4\") pod \"cluster-proxy-proxy-agent-c74dddb88-5dq45\" (UID: \"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:12:18.556448 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.556332 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e9c08cb-38ff-4ccb-9759-79bc336080fc-tmp\") pod \"klusterlet-addon-workmgr-5b7cbc654d-2xhcr\" (UID: \"4e9c08cb-38ff-4ccb-9759-79bc336080fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" Jul 15 12:12:18.556448 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.556350 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4e9c08cb-38ff-4ccb-9759-79bc336080fc-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b7cbc654d-2xhcr\" (UID: \"4e9c08cb-38ff-4ccb-9759-79bc336080fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" Jul 15 12:12:18.556448 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.556378 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkg9s\" (UniqueName: \"kubernetes.io/projected/4e9c08cb-38ff-4ccb-9759-79bc336080fc-kube-api-access-lkg9s\") pod \"klusterlet-addon-workmgr-5b7cbc654d-2xhcr\" (UID: \"4e9c08cb-38ff-4ccb-9759-79bc336080fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" Jul 15 12:12:18.556448 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.556395 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c822d778-a0fc-4d65-8ba4-d78dc9a11ca2-hub\") pod \"cluster-proxy-proxy-agent-c74dddb88-5dq45\" (UID: \"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:12:18.556448 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.556410 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c822d778-a0fc-4d65-8ba4-d78dc9a11ca2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c74dddb88-5dq45\" (UID: \"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:12:18.556448 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.556443 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35498a7e-03ff-41dd-bbb7-b34d3e4bc70e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75b8779674-g29jj\" (UID: \"35498a7e-03ff-41dd-bbb7-b34d3e4bc70e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b8779674-g29jj" Jul 15 12:12:18.556658 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.556457 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c822d778-a0fc-4d65-8ba4-d78dc9a11ca2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-c74dddb88-5dq45\" (UID: \"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:12:18.556658 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.556530 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xz6n\" (UniqueName: \"kubernetes.io/projected/35498a7e-03ff-41dd-bbb7-b34d3e4bc70e-kube-api-access-5xz6n\") pod \"managed-serviceaccount-addon-agent-75b8779674-g29jj\" (UID: \"35498a7e-03ff-41dd-bbb7-b34d3e4bc70e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b8779674-g29jj" Jul 15 12:12:18.656825 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.656738 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xz6n\" (UniqueName: \"kubernetes.io/projected/35498a7e-03ff-41dd-bbb7-b34d3e4bc70e-kube-api-access-5xz6n\") pod \"managed-serviceaccount-addon-agent-75b8779674-g29jj\" (UID: \"35498a7e-03ff-41dd-bbb7-b34d3e4bc70e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b8779674-g29jj" Jul 15 12:12:18.656825 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.656790 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c822d778-a0fc-4d65-8ba4-d78dc9a11ca2-ca\") pod \"cluster-proxy-proxy-agent-c74dddb88-5dq45\" (UID: \"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:12:18.656825 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.656805 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c822d778-a0fc-4d65-8ba4-d78dc9a11ca2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c74dddb88-5dq45\" (UID: \"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:12:18.657015 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.656840 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr6x4\" (UniqueName: \"kubernetes.io/projected/c822d778-a0fc-4d65-8ba4-d78dc9a11ca2-kube-api-access-mr6x4\") pod \"cluster-proxy-proxy-agent-c74dddb88-5dq45\" (UID: \"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:12:18.657015 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.656862 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e9c08cb-38ff-4ccb-9759-79bc336080fc-tmp\") pod \"klusterlet-addon-workmgr-5b7cbc654d-2xhcr\" (UID: \"4e9c08cb-38ff-4ccb-9759-79bc336080fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" Jul 15 12:12:18.657015 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.656880 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4e9c08cb-38ff-4ccb-9759-79bc336080fc-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b7cbc654d-2xhcr\" (UID: \"4e9c08cb-38ff-4ccb-9759-79bc336080fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" Jul 15 12:12:18.657015 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.656918 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkg9s\" (UniqueName: \"kubernetes.io/projected/4e9c08cb-38ff-4ccb-9759-79bc336080fc-kube-api-access-lkg9s\") pod \"klusterlet-addon-workmgr-5b7cbc654d-2xhcr\" (UID: \"4e9c08cb-38ff-4ccb-9759-79bc336080fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" Jul 15 12:12:18.657203 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.657058 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c822d778-a0fc-4d65-8ba4-d78dc9a11ca2-hub\") pod \"cluster-proxy-proxy-agent-c74dddb88-5dq45\" (UID: \"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:12:18.657203 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.657083 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c822d778-a0fc-4d65-8ba4-d78dc9a11ca2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c74dddb88-5dq45\" (UID: \"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:12:18.657203 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.657126 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35498a7e-03ff-41dd-bbb7-b34d3e4bc70e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75b8779674-g29jj\" (UID: \"35498a7e-03ff-41dd-bbb7-b34d3e4bc70e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b8779674-g29jj" Jul 15 12:12:18.657203 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.657152 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c822d778-a0fc-4d65-8ba4-d78dc9a11ca2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-c74dddb88-5dq45\" (UID: \"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:12:18.657399 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.657251 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e9c08cb-38ff-4ccb-9759-79bc336080fc-tmp\") pod \"klusterlet-addon-workmgr-5b7cbc654d-2xhcr\" (UID: \"4e9c08cb-38ff-4ccb-9759-79bc336080fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" Jul 15 12:12:18.657882 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.657854 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c822d778-a0fc-4d65-8ba4-d78dc9a11ca2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-c74dddb88-5dq45\" (UID: \"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:12:18.660995 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.660971 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c822d778-a0fc-4d65-8ba4-d78dc9a11ca2-ca\") pod \"cluster-proxy-proxy-agent-c74dddb88-5dq45\" (UID: \"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:12:18.660995 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.660985 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c822d778-a0fc-4d65-8ba4-d78dc9a11ca2-hub\") pod \"cluster-proxy-proxy-agent-c74dddb88-5dq45\" (UID: \"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:12:18.661142 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.661002 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35498a7e-03ff-41dd-bbb7-b34d3e4bc70e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75b8779674-g29jj\" (UID: \"35498a7e-03ff-41dd-bbb7-b34d3e4bc70e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b8779674-g29jj" Jul 15 12:12:18.661142 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.661025 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c822d778-a0fc-4d65-8ba4-d78dc9a11ca2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c74dddb88-5dq45\" (UID: \"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:12:18.661206 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.661145 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4e9c08cb-38ff-4ccb-9759-79bc336080fc-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b7cbc654d-2xhcr\" (UID: \"4e9c08cb-38ff-4ccb-9759-79bc336080fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" Jul 15 12:12:18.661525 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.661500 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c822d778-a0fc-4d65-8ba4-d78dc9a11ca2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c74dddb88-5dq45\" (UID: \"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:12:18.665285 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.665258 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkg9s\" (UniqueName: \"kubernetes.io/projected/4e9c08cb-38ff-4ccb-9759-79bc336080fc-kube-api-access-lkg9s\") pod \"klusterlet-addon-workmgr-5b7cbc654d-2xhcr\" (UID: \"4e9c08cb-38ff-4ccb-9759-79bc336080fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" Jul 15 12:12:18.666439 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.666416 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr6x4\" (UniqueName: \"kubernetes.io/projected/c822d778-a0fc-4d65-8ba4-d78dc9a11ca2-kube-api-access-mr6x4\") pod \"cluster-proxy-proxy-agent-c74dddb88-5dq45\" (UID: \"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:12:18.666561 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.666545 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xz6n\" (UniqueName: \"kubernetes.io/projected/35498a7e-03ff-41dd-bbb7-b34d3e4bc70e-kube-api-access-5xz6n\") pod \"managed-serviceaccount-addon-agent-75b8779674-g29jj\" (UID: \"35498a7e-03ff-41dd-bbb7-b34d3e4bc70e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b8779674-g29jj" Jul 15 12:12:18.695573 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.695542 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" Jul 15 12:12:18.729878 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.729847 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b8779674-g29jj" Jul 15 12:12:18.734699 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.734677 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:12:18.856357 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.856323 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr"] Jul 15 12:12:18.860119 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:12:18.859807 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9c08cb_38ff_4ccb_9759_79bc336080fc.slice/crio-ff67e6144e6affcbfc6778acfadd5dc2ebbabae1cc471d8757306c59eb956970 WatchSource:0}: Error finding container ff67e6144e6affcbfc6778acfadd5dc2ebbabae1cc471d8757306c59eb956970: Status 404 returned error can't find the container with id ff67e6144e6affcbfc6778acfadd5dc2ebbabae1cc471d8757306c59eb956970 Jul 15 12:12:18.890472 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.890439 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b8779674-g29jj"] Jul 15 12:12:18.892895 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:12:18.892872 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35498a7e_03ff_41dd_bbb7_b34d3e4bc70e.slice/crio-48c9cc2556d9f9d2bb143233a4e44d2ed59dc902e932cc51fa7de0ab3c360e71 WatchSource:0}: Error finding container 48c9cc2556d9f9d2bb143233a4e44d2ed59dc902e932cc51fa7de0ab3c360e71: Status 404 returned error can't find the container with id 48c9cc2556d9f9d2bb143233a4e44d2ed59dc902e932cc51fa7de0ab3c360e71 Jul 15 12:12:18.905869 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:18.905846 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45"] Jul 15 12:12:18.908375 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:12:18.908326 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc822d778_a0fc_4d65_8ba4_d78dc9a11ca2.slice/crio-693cd5b075e21888b407380b7f4c47d10ac6c013f31fc99a30ce289a9b3266ea WatchSource:0}: Error finding container 693cd5b075e21888b407380b7f4c47d10ac6c013f31fc99a30ce289a9b3266ea: Status 404 returned error can't find the container with id 693cd5b075e21888b407380b7f4c47d10ac6c013f31fc99a30ce289a9b3266ea Jul 15 12:12:19.095046 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:19.095015 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b8779674-g29jj" event={"ID":"35498a7e-03ff-41dd-bbb7-b34d3e4bc70e","Type":"ContainerStarted","Data":"48c9cc2556d9f9d2bb143233a4e44d2ed59dc902e932cc51fa7de0ab3c360e71"} Jul 15 12:12:19.096086 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:19.096061 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" event={"ID":"4e9c08cb-38ff-4ccb-9759-79bc336080fc","Type":"ContainerStarted","Data":"ff67e6144e6affcbfc6778acfadd5dc2ebbabae1cc471d8757306c59eb956970"} Jul 15 12:12:19.097022 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:19.097003 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" event={"ID":"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2","Type":"ContainerStarted","Data":"693cd5b075e21888b407380b7f4c47d10ac6c013f31fc99a30ce289a9b3266ea"} Jul 15 12:12:21.481916 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:21.481437 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs\") pod \"network-metrics-daemon-kfbnz\" (UID: \"e79deb5e-778c-420d-a65d-671a7d3d28db\") " pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:12:21.484969 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:21.484735 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 12:12:21.492345 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:12:21.492206 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 15 12:12:21.492345 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:12:21.492287 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs podName:e79deb5e-778c-420d-a65d-671a7d3d28db nodeName:}" failed. No retries permitted until 2026-07-15 12:13:25.492262462 +0000 UTC m=+129.126657392 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs") pod "network-metrics-daemon-kfbnz" (UID: "e79deb5e-778c-420d-a65d-671a7d3d28db") : secret "metrics-daemon-secret" not found Jul 15 12:12:21.582240 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:21.582075 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret\") pod \"global-pull-secret-syncer-sj4hd\" (UID: \"24a77a8a-6ec5-4558-8ce8-96ae7f4c9547\") " pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:12:21.585136 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:21.584928 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 12:12:21.600992 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:21.600130 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24a77a8a-6ec5-4558-8ce8-96ae7f4c9547-original-pull-secret\") pod \"global-pull-secret-syncer-sj4hd\" (UID: \"24a77a8a-6ec5-4558-8ce8-96ae7f4c9547\") " pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:12:21.682865 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:21.682820 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqmtr\" (UniqueName: \"kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr\") pod \"network-check-target-hqq7p\" (UID: \"ad5f3fa8-e4d3-42bf-8b16-94961009814d\") " pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:12:21.686360 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:21.686142 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 12:12:21.695937 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:21.695894 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 12:12:21.714010 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:21.713950 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqmtr\" (UniqueName: \"kubernetes.io/projected/ad5f3fa8-e4d3-42bf-8b16-94961009814d-kube-api-access-mqmtr\") pod \"network-check-target-hqq7p\" (UID: \"ad5f3fa8-e4d3-42bf-8b16-94961009814d\") " pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:12:21.859427 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:21.859391 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4hd" Jul 15 12:12:21.971380 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:21.971350 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tcn8s\"" Jul 15 12:12:21.979786 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:21.979742 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:12:22.627614 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:22.627502 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sj4hd"] Jul 15 12:12:22.630556 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:12:22.630528 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a77a8a_6ec5_4558_8ce8_96ae7f4c9547.slice/crio-b5bc69b90b5936e46f1eddd6003bb166de01fd4fdfd940c66e62fa07c86d96aa WatchSource:0}: Error finding container b5bc69b90b5936e46f1eddd6003bb166de01fd4fdfd940c66e62fa07c86d96aa: Status 404 returned error can't find the container with id b5bc69b90b5936e46f1eddd6003bb166de01fd4fdfd940c66e62fa07c86d96aa Jul 15 12:12:22.639832 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:22.639793 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hqq7p"] Jul 15 12:12:22.645101 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:12:22.645079 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5f3fa8_e4d3_42bf_8b16_94961009814d.slice/crio-eeb2188daccb2bd3b10cec9a8cd47505672d614c0afebe0366161f6848770f86 WatchSource:0}: Error finding container eeb2188daccb2bd3b10cec9a8cd47505672d614c0afebe0366161f6848770f86: Status 404 returned error can't find the container with id eeb2188daccb2bd3b10cec9a8cd47505672d614c0afebe0366161f6848770f86 Jul 15 12:12:23.122567 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:23.122533 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b8779674-g29jj" event={"ID":"35498a7e-03ff-41dd-bbb7-b34d3e4bc70e","Type":"ContainerStarted","Data":"e9be51a02ccffa0e1fb6d68a4dde853b6ae01f30425362725a2d21a9dbfa3b92"} Jul 15 12:12:23.124129 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:23.124092 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hqq7p" event={"ID":"ad5f3fa8-e4d3-42bf-8b16-94961009814d","Type":"ContainerStarted","Data":"eeb2188daccb2bd3b10cec9a8cd47505672d614c0afebe0366161f6848770f86"} Jul 15 12:12:23.125403 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:23.125378 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sj4hd" event={"ID":"24a77a8a-6ec5-4558-8ce8-96ae7f4c9547","Type":"ContainerStarted","Data":"b5bc69b90b5936e46f1eddd6003bb166de01fd4fdfd940c66e62fa07c86d96aa"} Jul 15 12:12:23.137710 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:23.137663 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b8779674-g29jj" podStartSLOduration=1.5087789649999999 podStartE2EDuration="5.137645971s" totalImagesPullingTime="3.628867006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:12:23.137071716 +0000 UTC m=+66.771466664" watchObservedRunningTime="2026-07-15 12:12:23.137645971 +0000 UTC m=+66.772040899" Jul 15 12:12:23.599654 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:23.599614 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:12:23.599852 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:12:23.599802 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 12:12:23.599903 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:12:23.599865 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls podName:7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9 nodeName:}" failed. No retries permitted until 2026-07-15 12:12:55.599849256 +0000 UTC m=+99.234244181 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls") pod "dns-default-b89tf" (UID: "7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9") : secret "dns-default-metrics-tls" not found Jul 15 12:12:26.147802 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:26.147005 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" event={"ID":"4e9c08cb-38ff-4ccb-9759-79bc336080fc","Type":"ContainerStarted","Data":"b09c7254ab20c97ce272c7b79747c93c4b05c74f3958fbaf0462fb61ab9c3550"} Jul 15 12:12:26.148276 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:26.147837 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" Jul 15 12:12:26.149503 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:26.149479 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" Jul 15 12:12:26.150237 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:26.150211 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" event={"ID":"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2","Type":"ContainerStarted","Data":"b19a2c9a546361dcce475a3f193affa9b06292e9afc787733b00302bc1504176"} Jul 15 12:12:26.150351 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:26.150250 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" event={"ID":"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2","Type":"ContainerStarted","Data":"22ef8656c6fd0af52ead28a236b40177cc1056424c0558a6d90955eec1b4d660"} Jul 15 12:12:26.182879 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:26.182821 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" podStartSLOduration=1.53915838 podStartE2EDuration="8.182801957s" totalImagesPullingTime="6.643643577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:12:26.165992808 +0000 UTC m=+69.800387757" watchObservedRunningTime="2026-07-15 12:12:26.182801957 +0000 UTC m=+69.817196906" Jul 15 12:12:28.158012 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:28.157978 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hqq7p" event={"ID":"ad5f3fa8-e4d3-42bf-8b16-94961009814d","Type":"ContainerStarted","Data":"04796357cb8935df4806abc83be2403d88bbbdc77ec365233d8b3f32071b3e3f"} Jul 15 12:12:28.158390 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:28.158060 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:12:28.160759 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:28.160728 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" event={"ID":"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2","Type":"ContainerStarted","Data":"38bd4a43460c009853edaf1c76150ecba6098b30e4c4e03333c0bfaf594139f5"} Jul 15 12:12:28.172484 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:28.172428 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hqq7p" podStartSLOduration=66.732399259 podStartE2EDuration="1m12.172407088s" totalImagesPullingTime="5.440007829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:12:28.171558899 +0000 UTC m=+71.805953846" watchObservedRunningTime="2026-07-15 12:12:28.172407088 +0000 UTC m=+71.806802038" Jul 15 12:12:28.189467 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:28.189410 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" podStartSLOduration=3.59187549 podStartE2EDuration="10.189395372s" totalImagesPullingTime="6.597519882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:12:28.18921692 +0000 UTC m=+71.823611896" watchObservedRunningTime="2026-07-15 12:12:28.189395372 +0000 UTC m=+71.823790320" Jul 15 12:12:29.164848 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:29.164805 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sj4hd" event={"ID":"24a77a8a-6ec5-4558-8ce8-96ae7f4c9547","Type":"ContainerStarted","Data":"e61dce5b18ba084facf2a5dfa463b35d1e1d6fc773c970025313cf2bc036de59"} Jul 15 12:12:29.179356 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:29.179313 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sj4hd" podStartSLOduration=67.72086933 podStartE2EDuration="1m13.179298113s" totalImagesPullingTime="5.458428783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:12:29.178088252 +0000 UTC m=+72.812483200" watchObservedRunningTime="2026-07-15 12:12:29.179298113 +0000 UTC m=+72.813693060" Jul 15 12:12:55.628403 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:55.628296 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:12:55.628924 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:12:55.628416 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 12:12:55.628924 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:12:55.628485 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls podName:7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9 nodeName:}" failed. No retries permitted until 2026-07-15 12:13:59.628469719 +0000 UTC m=+163.262864646 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls") pod "dns-default-b89tf" (UID: "7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9") : secret "dns-default-metrics-tls" not found Jul 15 12:12:59.167202 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:12:59.167167 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hqq7p" Jul 15 12:13:25.533826 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:13:25.533761 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs\") pod \"network-metrics-daemon-kfbnz\" (UID: \"e79deb5e-778c-420d-a65d-671a7d3d28db\") " pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:13:25.534296 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:13:25.533927 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 15 12:13:25.534296 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:13:25.533998 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs podName:e79deb5e-778c-420d-a65d-671a7d3d28db nodeName:}" failed. No retries permitted until 2026-07-15 12:15:27.533980762 +0000 UTC m=+251.168375692 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs") pod "network-metrics-daemon-kfbnz" (UID: "e79deb5e-778c-420d-a65d-671a7d3d28db") : secret "metrics-daemon-secret" not found Jul 15 12:13:54.788837 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:13:54.788790 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-b89tf" podUID="7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9" Jul 15 12:13:55.408472 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:13:55.408432 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b89tf" Jul 15 12:13:55.862377 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:13:55.862329 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kfbnz" podUID="e79deb5e-778c-420d-a65d-671a7d3d28db" Jul 15 12:13:59.656153 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:13:59.656113 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:13:59.656534 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:13:59.656250 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 12:13:59.656534 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:13:59.656321 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls podName:7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9 nodeName:}" failed. No retries permitted until 2026-07-15 12:16:01.656304991 +0000 UTC m=+285.290699917 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls") pod "dns-default-b89tf" (UID: "7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9") : secret "dns-default-metrics-tls" not found Jul 15 12:14:10.849043 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:14:10.848965 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:14:23.489178 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:14:23.489145 2556 generic.go:356] "Generic (PLEG): container finished" podID="35498a7e-03ff-41dd-bbb7-b34d3e4bc70e" containerID="e9be51a02ccffa0e1fb6d68a4dde853b6ae01f30425362725a2d21a9dbfa3b92" exitCode=255 Jul 15 12:14:23.489568 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:14:23.489218 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b8779674-g29jj" event={"ID":"35498a7e-03ff-41dd-bbb7-b34d3e4bc70e","Type":"ContainerDied","Data":"e9be51a02ccffa0e1fb6d68a4dde853b6ae01f30425362725a2d21a9dbfa3b92"} Jul 15 12:14:23.489568 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:14:23.489490 2556 scope.go:117] "RemoveContainer" containerID="e9be51a02ccffa0e1fb6d68a4dde853b6ae01f30425362725a2d21a9dbfa3b92" Jul 15 12:14:24.494143 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:14:24.494105 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b8779674-g29jj" event={"ID":"35498a7e-03ff-41dd-bbb7-b34d3e4bc70e","Type":"ContainerStarted","Data":"88cfde1fb6ce30a4fc5454fa49f92e231d8e12241a9beb428765d8c5e9fea762"} Jul 15 12:14:26.149036 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:14:26.148980 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" podUID="4e9c08cb-38ff-4ccb-9759-79bc336080fc" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.8:8000/readyz\": dial tcp 10.134.0.8:8000: connect: connection refused" Jul 15 12:14:26.501662 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:14:26.501631 2556 generic.go:356] "Generic (PLEG): container finished" podID="4e9c08cb-38ff-4ccb-9759-79bc336080fc" containerID="b09c7254ab20c97ce272c7b79747c93c4b05c74f3958fbaf0462fb61ab9c3550" exitCode=1 Jul 15 12:14:26.501833 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:14:26.501701 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" event={"ID":"4e9c08cb-38ff-4ccb-9759-79bc336080fc","Type":"ContainerDied","Data":"b09c7254ab20c97ce272c7b79747c93c4b05c74f3958fbaf0462fb61ab9c3550"} Jul 15 12:14:26.502042 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:14:26.502029 2556 scope.go:117] "RemoveContainer" containerID="b09c7254ab20c97ce272c7b79747c93c4b05c74f3958fbaf0462fb61ab9c3550" Jul 15 12:14:27.505806 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:14:27.505754 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" event={"ID":"4e9c08cb-38ff-4ccb-9759-79bc336080fc","Type":"ContainerStarted","Data":"2ba2b5d0c6647d75353223b8d6313ee6e687f95dea92a7cb2522d2ee7db5b634"} Jul 15 12:14:27.506176 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:14:27.506054 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" Jul 15 12:14:27.506653 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:14:27.506636 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cbc654d-2xhcr" Jul 15 12:14:58.266133 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:14:58.266089 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:58.355353 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:14:58.355317 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:00.339680 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:00.339638 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:00.390250 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:00.390211 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:03.417025 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.416992 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nlxxv"] Jul 15 12:15:03.419391 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.419368 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:03.422208 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.422187 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 12:15:03.423464 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.423445 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 12:15:03.423551 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.423471 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jn585\"" Jul 15 12:15:03.423551 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.423514 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 12:15:03.423551 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.423449 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 12:15:03.433294 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.433264 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nlxxv"] Jul 15 12:15:03.442311 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.442290 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:03.471952 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.471932 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cceacc62-64ef-49d1-baac-a24bab3c479c-crio-socket\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:03.472065 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.471959 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:03.472065 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.471991 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cceacc62-64ef-49d1-baac-a24bab3c479c-data-volume\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:03.472065 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.472007 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:03.472160 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.472064 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85g68\" (UniqueName: \"kubernetes.io/projected/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-api-access-85g68\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:03.501090 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.501066 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:03.572936 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.572914 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cceacc62-64ef-49d1-baac-a24bab3c479c-crio-socket\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:03.573042 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.572942 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:03.573042 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.572972 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cceacc62-64ef-49d1-baac-a24bab3c479c-data-volume\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:03.573042 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.572994 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:03.573042 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.573028 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cceacc62-64ef-49d1-baac-a24bab3c479c-crio-socket\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:03.573242 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:03.573056 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 12:15:03.573242 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:03.573088 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 12:15:03.573242 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:03.573107 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-rbac-proxy-cm podName:cceacc62-64ef-49d1-baac-a24bab3c479c nodeName:}" failed. No retries permitted until 2026-07-15 12:15:04.073093065 +0000 UTC m=+227.707487991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nlxxv" (UID: "cceacc62-64ef-49d1-baac-a24bab3c479c") : configmap "kube-rbac-proxy" not found Jul 15 12:15:03.573242 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.573143 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85g68\" (UniqueName: \"kubernetes.io/projected/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-api-access-85g68\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:03.573242 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:03.573161 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls podName:cceacc62-64ef-49d1-baac-a24bab3c479c nodeName:}" failed. No retries permitted until 2026-07-15 12:15:04.073141479 +0000 UTC m=+227.707536419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nlxxv" (UID: "cceacc62-64ef-49d1-baac-a24bab3c479c") : secret "insights-runtime-extractor-tls" not found Jul 15 12:15:03.573419 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.573259 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cceacc62-64ef-49d1-baac-a24bab3c479c-data-volume\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:03.584114 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:03.584088 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85g68\" (UniqueName: \"kubernetes.io/projected/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-api-access-85g68\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:04.076590 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:04.076557 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:04.076590 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:04.076598 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:04.077023 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:04.076670 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 12:15:04.077023 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:04.076708 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 12:15:04.077023 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:04.076736 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-rbac-proxy-cm podName:cceacc62-64ef-49d1-baac-a24bab3c479c nodeName:}" failed. No retries permitted until 2026-07-15 12:15:05.076716731 +0000 UTC m=+228.711111659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nlxxv" (UID: "cceacc62-64ef-49d1-baac-a24bab3c479c") : configmap "kube-rbac-proxy" not found Jul 15 12:15:04.077023 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:04.076753 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls podName:cceacc62-64ef-49d1-baac-a24bab3c479c nodeName:}" failed. No retries permitted until 2026-07-15 12:15:05.076742547 +0000 UTC m=+228.711137496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nlxxv" (UID: "cceacc62-64ef-49d1-baac-a24bab3c479c") : secret "insights-runtime-extractor-tls" not found Jul 15 12:15:04.443083 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:04.443005 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:04.543530 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:04.543500 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:04.644603 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:04.644564 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:04.692032 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:04.692004 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:04.850260 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:04.850230 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:04.901349 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:04.901320 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:05.082527 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:05.082487 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:05.082527 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:05.082534 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:05.082729 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:05.082616 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 12:15:05.082729 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:05.082645 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 12:15:05.082729 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:05.082670 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-rbac-proxy-cm podName:cceacc62-64ef-49d1-baac-a24bab3c479c nodeName:}" failed. No retries permitted until 2026-07-15 12:15:07.082655816 +0000 UTC m=+230.717050742 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nlxxv" (UID: "cceacc62-64ef-49d1-baac-a24bab3c479c") : configmap "kube-rbac-proxy" not found Jul 15 12:15:05.082729 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:05.082699 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls podName:cceacc62-64ef-49d1-baac-a24bab3c479c nodeName:}" failed. No retries permitted until 2026-07-15 12:15:07.082680661 +0000 UTC m=+230.717075587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nlxxv" (UID: "cceacc62-64ef-49d1-baac-a24bab3c479c") : secret "insights-runtime-extractor-tls" not found Jul 15 12:15:05.185636 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:05.185567 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b8nd7_a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e/dns-node-resolver/0.log" Jul 15 12:15:05.331959 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:05.331929 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:05.439107 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:05.439027 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:05.530449 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:05.530416 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:05.585796 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:05.585747 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:06.585056 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:06.585035 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p8nqm_6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8/node-ca/0.log" Jul 15 12:15:07.024996 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:07.024960 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:07.079437 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:07.079400 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:07.097393 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:07.097370 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:07.097479 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:07.097405 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:07.097516 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:07.097485 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 12:15:07.097549 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:07.097528 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-rbac-proxy-cm podName:cceacc62-64ef-49d1-baac-a24bab3c479c nodeName:}" failed. No retries permitted until 2026-07-15 12:15:11.097515795 +0000 UTC m=+234.731910720 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nlxxv" (UID: "cceacc62-64ef-49d1-baac-a24bab3c479c") : configmap "kube-rbac-proxy" not found Jul 15 12:15:07.097628 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:07.097613 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 12:15:07.097668 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:07.097663 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls podName:cceacc62-64ef-49d1-baac-a24bab3c479c nodeName:}" failed. No retries permitted until 2026-07-15 12:15:11.097650461 +0000 UTC m=+234.732045387 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nlxxv" (UID: "cceacc62-64ef-49d1-baac-a24bab3c479c") : secret "insights-runtime-extractor-tls" not found Jul 15 12:15:07.185213 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:07.185183 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fb9dn_989c12ef-5656-4c13-be8d-84a41e5ab36b/serve-healthcheck-canary/0.log" Jul 15 12:15:07.829058 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:07.829016 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:11.125551 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:11.125507 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:11.125551 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:11.125557 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:11.126004 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:11.125646 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 12:15:11.126004 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:11.125656 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 12:15:11.126004 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:11.125704 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-rbac-proxy-cm podName:cceacc62-64ef-49d1-baac-a24bab3c479c nodeName:}" failed. No retries permitted until 2026-07-15 12:15:19.125688846 +0000 UTC m=+242.760083772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nlxxv" (UID: "cceacc62-64ef-49d1-baac-a24bab3c479c") : configmap "kube-rbac-proxy" not found Jul 15 12:15:11.126004 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:11.125716 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls podName:cceacc62-64ef-49d1-baac-a24bab3c479c nodeName:}" failed. No retries permitted until 2026-07-15 12:15:19.125710916 +0000 UTC m=+242.760105842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nlxxv" (UID: "cceacc62-64ef-49d1-baac-a24bab3c479c") : secret "insights-runtime-extractor-tls" not found Jul 15 12:15:11.158746 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:11.158713 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:11.928068 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:11.928032 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:12.049320 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:12.049278 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:12.293474 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:12.293434 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:12.402604 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:12.402565 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:13.024598 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:13.024565 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:13.626347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:13.626274 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:13.825707 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:13.825673 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:14.220231 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:14.220181 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:14.270083 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:14.270052 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:14.457346 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:14.457316 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:14.828902 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:14.828865 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:15.269736 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:15.269700 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:15.422562 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:15.422533 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:15.826330 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:15.826296 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:15.900543 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:15.900510 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:16.027627 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:16.027593 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:16.625068 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:16.625039 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:16.675357 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:16.675321 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:17.230791 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:17.230745 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:17.430241 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:17.430199 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:17.823023 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:17.822987 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:18.026502 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:18.026462 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:18.429688 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:18.429657 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:18.487985 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:18.487949 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:19.053867 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:19.053818 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:19.183235 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:19.183207 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:19.183380 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:19.183246 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:19.183380 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:19.183357 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 12:15:19.183467 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:19.183429 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls podName:cceacc62-64ef-49d1-baac-a24bab3c479c nodeName:}" failed. No retries permitted until 2026-07-15 12:15:35.183413424 +0000 UTC m=+258.817808349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nlxxv" (UID: "cceacc62-64ef-49d1-baac-a24bab3c479c") : secret "insights-runtime-extractor-tls" not found Jul 15 12:15:19.183689 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:19.183673 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cceacc62-64ef-49d1-baac-a24bab3c479c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:19.425523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:19.425447 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:19.623739 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:19.623709 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:20.027963 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:20.027927 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:20.627637 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:20.627604 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:21.024521 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:21.024488 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:21.232860 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:21.232823 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:21.835736 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:21.835701 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:22.294252 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:22.294217 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:22.435509 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:22.435474 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:22.625204 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:22.625109 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:27.543718 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:27.543676 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs\") pod \"network-metrics-daemon-kfbnz\" (UID: \"e79deb5e-778c-420d-a65d-671a7d3d28db\") " pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:15:27.546048 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:27.546026 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e79deb5e-778c-420d-a65d-671a7d3d28db-metrics-certs\") pod \"network-metrics-daemon-kfbnz\" (UID: \"e79deb5e-778c-420d-a65d-671a7d3d28db\") " pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:15:27.652009 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:27.651977 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5qgtr\"" Jul 15 12:15:27.660032 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:27.660011 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfbnz" Jul 15 12:15:27.782093 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:27.782038 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kfbnz"] Jul 15 12:15:27.785755 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:15:27.785727 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79deb5e_778c_420d_a65d_671a7d3d28db.slice/crio-2d98ef5cd72771de45234d1e385c85669d975054dc1e6956f135120665856a96 WatchSource:0}: Error finding container 2d98ef5cd72771de45234d1e385c85669d975054dc1e6956f135120665856a96: Status 404 returned error can't find the container with id 2d98ef5cd72771de45234d1e385c85669d975054dc1e6956f135120665856a96 Jul 15 12:15:28.680742 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:28.680710 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kfbnz" event={"ID":"e79deb5e-778c-420d-a65d-671a7d3d28db","Type":"ContainerStarted","Data":"2d98ef5cd72771de45234d1e385c85669d975054dc1e6956f135120665856a96"} Jul 15 12:15:28.722264 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:28.722226 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:28.951489 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:28.951390 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:29.685723 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:29.685677 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kfbnz" event={"ID":"e79deb5e-778c-420d-a65d-671a7d3d28db","Type":"ContainerStarted","Data":"222234fab047050bb54ed32b1d721c8a0374af843227a9fa1a9c110ace2c7fa6"} Jul 15 12:15:29.686102 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:29.685802 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kfbnz" event={"ID":"e79deb5e-778c-420d-a65d-671a7d3d28db","Type":"ContainerStarted","Data":"051aab7e93c0cc5bc159db64e5ddfc0381d5f3a86652a74241ba920d676f92c7"} Jul 15 12:15:29.702081 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:29.702031 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kfbnz" podStartSLOduration=252.337032361 podStartE2EDuration="4m13.702017295s" totalImagesPullingTime="1.364984934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:15:29.701619571 +0000 UTC m=+253.336014519" watchObservedRunningTime="2026-07-15 12:15:29.702017295 +0000 UTC m=+253.336412240" Jul 15 12:15:30.267827 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:30.267797 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:30.359765 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:30.359713 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:30.430422 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:30.430384 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:32.580560 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:32.580519 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:32.678492 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:32.678466 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:32.747126 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:32.747094 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:33.081659 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:33.081623 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:35.196974 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:35.196939 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:35.199247 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:35.199227 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cceacc62-64ef-49d1-baac-a24bab3c479c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlxxv\" (UID: \"cceacc62-64ef-49d1-baac-a24bab3c479c\") " pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:35.230921 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:35.230892 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jn585\"" Jul 15 12:15:35.239258 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:35.239226 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nlxxv" Jul 15 12:15:35.364216 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:35.364071 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nlxxv"] Jul 15 12:15:35.366146 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:15:35.366119 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcceacc62_64ef_49d1_baac_a24bab3c479c.slice/crio-4f295a18884e1fe19e7181700fb810b2116a102025108802cb04c54547a9b915 WatchSource:0}: Error finding container 4f295a18884e1fe19e7181700fb810b2116a102025108802cb04c54547a9b915: Status 404 returned error can't find the container with id 4f295a18884e1fe19e7181700fb810b2116a102025108802cb04c54547a9b915 Jul 15 12:15:35.703415 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:35.703370 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nlxxv" event={"ID":"cceacc62-64ef-49d1-baac-a24bab3c479c","Type":"ContainerStarted","Data":"4f295a18884e1fe19e7181700fb810b2116a102025108802cb04c54547a9b915"} Jul 15 12:15:35.763627 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:35.763595 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:35.835905 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:35.835864 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:36.370679 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:36.370622 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:40.722181 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:40.722147 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nlxxv" event={"ID":"cceacc62-64ef-49d1-baac-a24bab3c479c","Type":"ContainerStarted","Data":"f2f07c4f1d50869a7c4d233e9c19223cd7f979f430a8c289a90435e491323c89"} Jul 15 12:15:41.292658 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:41.292554 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:42.906598 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:42.906565 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:43.048282 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.048193 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:43.151933 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.151896 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:43.210337 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.210265 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m89t4"] Jul 15 12:15:43.228733 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.228700 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.231510 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.231483 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 12:15:43.231655 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.231489 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 12:15:43.231655 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.231487 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 12:15:43.232539 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.232520 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 12:15:43.232643 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.232561 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 12:15:43.232699 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.232525 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 12:15:43.232882 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.232861 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n2sgv\"" Jul 15 12:15:43.253970 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.253925 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:43.310338 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.310310 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:43.360283 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.360256 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-sys\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.360439 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.360299 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-node-exporter-tls\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.360439 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.360324 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfxmj\" (UniqueName: \"kubernetes.io/projected/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-kube-api-access-lfxmj\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.360439 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.360391 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-metrics-client-ca\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.360439 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.360428 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-node-exporter-textfile\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.360574 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.360464 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-root\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.360574 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.360480 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-node-exporter-accelerators-collector-config\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.360574 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.360559 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.360662 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.360583 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-node-exporter-wtmp\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.414193 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.414148 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:43.461175 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.461087 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-sys\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.461175 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.461143 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-node-exporter-tls\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.461373 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.461181 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfxmj\" (UniqueName: \"kubernetes.io/projected/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-kube-api-access-lfxmj\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.461373 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.461214 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-metrics-client-ca\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.461373 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.461212 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-sys\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.461373 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.461248 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-node-exporter-textfile\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.461373 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.461288 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-root\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.461373 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.461311 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-node-exporter-accelerators-collector-config\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.461373 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.461349 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.461625 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.461390 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-node-exporter-wtmp\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.461625 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.461538 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-node-exporter-wtmp\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.461625 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.461562 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-root\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.462122 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.462028 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-node-exporter-textfile\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.462242 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.462185 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-node-exporter-accelerators-collector-config\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.462242 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.462209 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-metrics-client-ca\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.463964 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.463936 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-node-exporter-tls\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.464078 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.464034 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.471360 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.471318 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfxmj\" (UniqueName: \"kubernetes.io/projected/e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c-kube-api-access-lfxmj\") pod \"node-exporter-m89t4\" (UID: \"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c\") " pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.515020 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.514990 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:43.538786 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.538740 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m89t4" Jul 15 12:15:43.618700 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.618661 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:43.719904 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:43.719817 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:44.013591 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:44.013560 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:44.028161 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:15:44.028138 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8432f3a_b5b7_47b2_a530_1d5c31bc8e1c.slice/crio-2096d420ec560c2dcd53e5d67d4dc12eb4030d43c96b6dc9695951d89d4e864b WatchSource:0}: Error finding container 2096d420ec560c2dcd53e5d67d4dc12eb4030d43c96b6dc9695951d89d4e864b: Status 404 returned error can't find the container with id 2096d420ec560c2dcd53e5d67d4dc12eb4030d43c96b6dc9695951d89d4e864b Jul 15 12:15:44.132327 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:44.132296 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:44.214842 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:44.214811 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:44.629990 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:44.629939 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:44.742400 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:44.742350 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nlxxv" event={"ID":"cceacc62-64ef-49d1-baac-a24bab3c479c","Type":"ContainerStarted","Data":"efcf231acf2fecf9a53c8dba6539d4dc8eb839bc0253033bed5c8553c0b819ba"} Jul 15 12:15:44.743661 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:44.743629 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m89t4" event={"ID":"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c","Type":"ContainerStarted","Data":"2096d420ec560c2dcd53e5d67d4dc12eb4030d43c96b6dc9695951d89d4e864b"} Jul 15 12:15:45.748823 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:45.748765 2556 generic.go:356] "Generic (PLEG): container finished" podID="e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c" containerID="922c60aa5dd5502033098641f68f2fdd4dbf2d93f7c519dcfc6d49bc168f26d1" exitCode=0 Jul 15 12:15:45.749275 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:45.748814 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m89t4" event={"ID":"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c","Type":"ContainerDied","Data":"922c60aa5dd5502033098641f68f2fdd4dbf2d93f7c519dcfc6d49bc168f26d1"} Jul 15 12:15:46.075722 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:46.075614 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:46.176917 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:46.176882 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:46.304692 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:46.304643 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:46.755086 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:46.755052 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m89t4" event={"ID":"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c","Type":"ContainerStarted","Data":"ac3e1bc473d9052fc6f652bcc63e95c9f6b693f9928bff1dedf81fe7d3a174bd"} Jul 15 12:15:46.755516 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:46.755112 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m89t4" event={"ID":"e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c","Type":"ContainerStarted","Data":"9f405e666c83fd5b6ff309495ea5bb5a75f769b21469c05763be7f9c4c1d7a00"} Jul 15 12:15:46.779444 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:46.779384 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m89t4" podStartSLOduration=2.294992345 podStartE2EDuration="3.779361626s" totalImagesPullingTime="1.403917274s" totalInitContainerRuntime="80.452007ms" isStatefulPod=true podCreationTimestamp="2026-07-15 12:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:15:46.777706701 +0000 UTC m=+270.412101672" watchObservedRunningTime="2026-07-15 12:15:46.779361626 +0000 UTC m=+270.413756574" Jul 15 12:15:46.807006 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:46.806972 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:47.760025 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:47.759991 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nlxxv" event={"ID":"cceacc62-64ef-49d1-baac-a24bab3c479c","Type":"ContainerStarted","Data":"9bbbf4b8a75ff37e2e813d361a67a3367805f1a48ea18b05ab5b80e407f3710e"} Jul 15 12:15:47.783767 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:47.783716 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nlxxv" podStartSLOduration=32.64925416 podStartE2EDuration="44.783702596s" totalImagesPullingTime="12.134448436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:15:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:15:47.78151184 +0000 UTC m=+271.415906789" watchObservedRunningTime="2026-07-15 12:15:47.783702596 +0000 UTC m=+271.418097601" Jul 15 12:15:47.921871 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:47.921790 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:48.199607 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:48.199523 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:49.330179 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:49.330132 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:51.418647 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:51.418603 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:57.337245 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:57.337207 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:58.409364 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:15:58.409327 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-b89tf" podUID="7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9" Jul 15 12:15:58.552347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:58.552313 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:58.716006 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:58.715912 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:58.793206 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:58.793179 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b89tf" Jul 15 12:15:58.816713 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:58.816677 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:58.868876 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:58.868842 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:59.134166 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:59.134137 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:59.232160 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:59.232130 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:59.432972 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:15:59.432877 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:00.624797 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:00.624740 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:01.693789 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:01.693740 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:16:01.696122 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:01.696098 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9-metrics-tls\") pod \"dns-default-b89tf\" (UID: \"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9\") " pod="openshift-dns/dns-default-b89tf" Jul 15 12:16:01.796769 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:01.796734 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gfjwd\"" Jul 15 12:16:01.804389 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:01.804366 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b89tf" Jul 15 12:16:01.928205 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:01.928174 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b89tf"] Jul 15 12:16:01.931281 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:16:01.931255 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7478a0e7_eeb9_4da2_8a4b_c7f5cde188b9.slice/crio-4cd3421fe56786be56aa0c29528c744282c7cb68233ca8dc71093b17c0b74f91 WatchSource:0}: Error finding container 4cd3421fe56786be56aa0c29528c744282c7cb68233ca8dc71093b17c0b74f91: Status 404 returned error can't find the container with id 4cd3421fe56786be56aa0c29528c744282c7cb68233ca8dc71093b17c0b74f91 Jul 15 12:16:02.109446 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:02.109418 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:02.158328 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:02.158293 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:02.478710 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:02.478612 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:02.568681 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:02.568640 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:02.621857 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:02.621821 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:02.808944 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:02.808906 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b89tf" event={"ID":"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9","Type":"ContainerStarted","Data":"4cd3421fe56786be56aa0c29528c744282c7cb68233ca8dc71093b17c0b74f91"} Jul 15 12:16:02.937269 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:02.937227 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:03.043524 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:03.043488 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:03.813901 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:03.813817 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b89tf" event={"ID":"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9","Type":"ContainerStarted","Data":"ca1c4a5ff5a4fbbb7125ce3f7a8b34333515ee5f02687891552075ff21987cec"} Jul 15 12:16:03.813901 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:03.813867 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b89tf" event={"ID":"7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9","Type":"ContainerStarted","Data":"890f4f963a93a534661628ef52f45c6d0d7bb6605444199a7d4e9f64c6446c14"} Jul 15 12:16:03.814284 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:03.813927 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b89tf" Jul 15 12:16:03.830739 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:03.830695 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-b89tf" podStartSLOduration=251.23335484 podStartE2EDuration="4m12.830682296s" totalImagesPullingTime="1.597327456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:11:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:16:03.829728125 +0000 UTC m=+287.464123072" watchObservedRunningTime="2026-07-15 12:16:03.830682296 +0000 UTC m=+287.465077243" Jul 15 12:16:05.446236 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:05.446182 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:05.498459 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:05.498429 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:08.736066 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:08.736023 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" podUID="c822d778-a0fc-4d65-8ba4-d78dc9a11ca2" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 15 12:16:13.025678 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:13.025624 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:13.721731 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:13.721697 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:13.776610 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:13.776583 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:13.819527 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:13.819497 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b89tf" Jul 15 12:16:16.699185 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:16.699155 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:16:16.699650 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:16.699501 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:16:16.706101 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:16.706081 2556 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 12:16:16.825613 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:16.825573 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:16.881329 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:16.881189 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:18.735842 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:18.735809 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" podUID="c822d778-a0fc-4d65-8ba4-d78dc9a11ca2" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 15 12:16:19.324570 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:19.324540 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:19.377740 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:19.377710 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:21.672711 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:21.672676 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:21.737513 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:21.737470 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:23.077024 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:23.076992 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:23.180311 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:23.180269 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:24.034480 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:24.034449 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:24.135136 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:24.135105 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:28.413112 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:28.413068 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:28.511107 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:28.511082 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:28.728364 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:28.728271 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:28.736851 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:28.736819 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" podUID="c822d778-a0fc-4d65-8ba4-d78dc9a11ca2" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 15 12:16:28.736991 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:28.736904 2556 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" Jul 15 12:16:28.737414 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:28.737378 2556 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"38bd4a43460c009853edaf1c76150ecba6098b30e4c4e03333c0bfaf594139f5"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 15 12:16:28.737529 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:28.737437 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" podUID="c822d778-a0fc-4d65-8ba4-d78dc9a11ca2" containerName="service-proxy" containerID="cri-o://38bd4a43460c009853edaf1c76150ecba6098b30e4c4e03333c0bfaf594139f5" gracePeriod=30 Jul 15 12:16:28.829739 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:28.829702 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:28.895818 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:28.895711 2556 generic.go:356] "Generic (PLEG): container finished" podID="c822d778-a0fc-4d65-8ba4-d78dc9a11ca2" containerID="38bd4a43460c009853edaf1c76150ecba6098b30e4c4e03333c0bfaf594139f5" exitCode=2 Jul 15 12:16:28.895818 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:28.895753 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" event={"ID":"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2","Type":"ContainerDied","Data":"38bd4a43460c009853edaf1c76150ecba6098b30e4c4e03333c0bfaf594139f5"} Jul 15 12:16:29.901548 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:29.901512 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c74dddb88-5dq45" event={"ID":"c822d778-a0fc-4d65-8ba4-d78dc9a11ca2","Type":"ContainerStarted","Data":"a8dca1357f506901cca0d05c4e8c211b9eb2ed2bdf8f0cbce49d773c74affab5"} Jul 15 12:16:30.684944 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:30.684910 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:30.740570 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:30.740536 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:45.844742 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:45.844703 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:45.903516 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:45.903419 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:50.927967 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:50.927931 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:59.204824 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:59.204794 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:59.257464 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:16:59.257431 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:00.403532 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:17:00.403490 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:00.542833 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:17:00.542803 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:00.599968 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:17:00.599939 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:01.149908 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:17:01.149873 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:08.813640 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:17:08.813544 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:09.187227 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:17:09.187136 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:11.227031 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:17:11.226992 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:11.277260 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:17:11.277232 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:14.646081 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:17:14.646035 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:14.693958 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:17:14.693923 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:28.752484 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:17:28.752448 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:28.852193 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:17:28.852159 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:03.408061 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:03.408025 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:03.509479 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:03.509449 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:04.534420 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:04.534386 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:04.587965 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:04.587931 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:07.064140 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:07.064100 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:18.495423 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:18.495359 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:20.075976 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:20.075933 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:30.037934 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:30.037901 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:30.823591 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:30.823556 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:33.648146 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:33.648107 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:33.702281 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:33.702248 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:34.757449 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:34.757414 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:34.861878 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:34.861847 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:35.767462 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:35.767424 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:35.852222 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:35.852187 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:37.921641 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:37.921552 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:38.349637 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:38.349606 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:38.390212 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:38.390182 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lkq7k"] Jul 15 12:18:38.391968 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:38.391954 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lkq7k" Jul 15 12:18:38.394649 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:38.394626 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 15 12:18:38.394875 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:38.394742 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8hz5n\"" Jul 15 12:18:38.394875 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:38.394794 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 15 12:18:38.401847 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:38.401827 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lkq7k"] Jul 15 12:18:38.421263 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:38.421236 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td7th\" (UniqueName: \"kubernetes.io/projected/40ccae27-9a83-448d-8033-eb2cee5874c9-kube-api-access-td7th\") pod \"cert-manager-cainjector-559cdd8c87-lkq7k\" (UID: \"40ccae27-9a83-448d-8033-eb2cee5874c9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lkq7k" Jul 15 12:18:38.421372 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:38.421276 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40ccae27-9a83-448d-8033-eb2cee5874c9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lkq7k\" (UID: \"40ccae27-9a83-448d-8033-eb2cee5874c9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lkq7k" Jul 15 12:18:38.521615 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:38.521579 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td7th\" (UniqueName: \"kubernetes.io/projected/40ccae27-9a83-448d-8033-eb2cee5874c9-kube-api-access-td7th\") pod \"cert-manager-cainjector-559cdd8c87-lkq7k\" (UID: \"40ccae27-9a83-448d-8033-eb2cee5874c9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lkq7k" Jul 15 12:18:38.521769 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:38.521636 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40ccae27-9a83-448d-8033-eb2cee5874c9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lkq7k\" (UID: \"40ccae27-9a83-448d-8033-eb2cee5874c9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lkq7k" Jul 15 12:18:38.529534 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:38.529506 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40ccae27-9a83-448d-8033-eb2cee5874c9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lkq7k\" (UID: \"40ccae27-9a83-448d-8033-eb2cee5874c9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lkq7k" Jul 15 12:18:38.529683 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:38.529666 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td7th\" (UniqueName: \"kubernetes.io/projected/40ccae27-9a83-448d-8033-eb2cee5874c9-kube-api-access-td7th\") pod \"cert-manager-cainjector-559cdd8c87-lkq7k\" (UID: \"40ccae27-9a83-448d-8033-eb2cee5874c9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lkq7k" Jul 15 12:18:38.700925 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:38.700847 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lkq7k" Jul 15 12:18:38.824327 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:38.824305 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lkq7k"] Jul 15 12:18:38.826834 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:18:38.826804 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40ccae27_9a83_448d_8033_eb2cee5874c9.slice/crio-fd32e2729f77c793b03e1f309b625c2ab7a337532e407b84011a1e3d98630bbb WatchSource:0}: Error finding container fd32e2729f77c793b03e1f309b625c2ab7a337532e407b84011a1e3d98630bbb: Status 404 returned error can't find the container with id fd32e2729f77c793b03e1f309b625c2ab7a337532e407b84011a1e3d98630bbb Jul 15 12:18:38.828485 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:38.828468 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 12:18:39.320735 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:39.320702 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lkq7k" event={"ID":"40ccae27-9a83-448d-8033-eb2cee5874c9","Type":"ContainerStarted","Data":"fd32e2729f77c793b03e1f309b625c2ab7a337532e407b84011a1e3d98630bbb"} Jul 15 12:18:40.362049 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:40.362003 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:40.466954 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:40.466916 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:42.333468 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:42.333431 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lkq7k" event={"ID":"40ccae27-9a83-448d-8033-eb2cee5874c9","Type":"ContainerStarted","Data":"df61d5f69c6c5bed88e3cc69997f2b5cfec80e3867291b5ec27cc41cfefcdd24"} Jul 15 12:18:42.351402 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:42.351358 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lkq7k" podStartSLOduration=1.343114043 podStartE2EDuration="4.351343623s" totalImagesPullingTime="3.00822958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:18:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:18:42.350214255 +0000 UTC m=+445.984609207" watchObservedRunningTime="2026-07-15 12:18:42.351343623 +0000 UTC m=+445.985738571" Jul 15 12:18:46.247598 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:46.247569 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:46.347360 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:46.347326 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:46.544404 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:46.544302 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:52.159570 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:52.159535 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:52.265338 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:52.265288 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:54.395040 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:54.395007 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:56.739683 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:56.739651 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-29vz4"] Jul 15 12:18:56.742760 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:56.742744 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-29vz4" Jul 15 12:18:56.745405 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:56.745388 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-hfx8k\"" Jul 15 12:18:56.752330 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:56.752308 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-29vz4"] Jul 15 12:18:56.841337 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:56.841307 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57ecd3c1-4b91-4730-b8da-9e55ae0204eb-bound-sa-token\") pod \"cert-manager-55b4d954d5-29vz4\" (UID: \"57ecd3c1-4b91-4730-b8da-9e55ae0204eb\") " pod="cert-manager/cert-manager-55b4d954d5-29vz4" Jul 15 12:18:56.841337 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:56.841339 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjtsl\" (UniqueName: \"kubernetes.io/projected/57ecd3c1-4b91-4730-b8da-9e55ae0204eb-kube-api-access-rjtsl\") pod \"cert-manager-55b4d954d5-29vz4\" (UID: \"57ecd3c1-4b91-4730-b8da-9e55ae0204eb\") " pod="cert-manager/cert-manager-55b4d954d5-29vz4" Jul 15 12:18:56.942075 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:56.942045 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57ecd3c1-4b91-4730-b8da-9e55ae0204eb-bound-sa-token\") pod \"cert-manager-55b4d954d5-29vz4\" (UID: \"57ecd3c1-4b91-4730-b8da-9e55ae0204eb\") " pod="cert-manager/cert-manager-55b4d954d5-29vz4" Jul 15 12:18:56.942075 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:56.942076 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjtsl\" (UniqueName: \"kubernetes.io/projected/57ecd3c1-4b91-4730-b8da-9e55ae0204eb-kube-api-access-rjtsl\") pod \"cert-manager-55b4d954d5-29vz4\" (UID: \"57ecd3c1-4b91-4730-b8da-9e55ae0204eb\") " pod="cert-manager/cert-manager-55b4d954d5-29vz4" Jul 15 12:18:56.950805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:56.950766 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjtsl\" (UniqueName: \"kubernetes.io/projected/57ecd3c1-4b91-4730-b8da-9e55ae0204eb-kube-api-access-rjtsl\") pod \"cert-manager-55b4d954d5-29vz4\" (UID: \"57ecd3c1-4b91-4730-b8da-9e55ae0204eb\") " pod="cert-manager/cert-manager-55b4d954d5-29vz4" Jul 15 12:18:56.950907 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:56.950807 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57ecd3c1-4b91-4730-b8da-9e55ae0204eb-bound-sa-token\") pod \"cert-manager-55b4d954d5-29vz4\" (UID: \"57ecd3c1-4b91-4730-b8da-9e55ae0204eb\") " pod="cert-manager/cert-manager-55b4d954d5-29vz4" Jul 15 12:18:57.051182 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:57.051142 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-29vz4" Jul 15 12:18:57.172686 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:57.172641 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-29vz4"] Jul 15 12:18:57.175046 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:18:57.175021 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57ecd3c1_4b91_4730_b8da_9e55ae0204eb.slice/crio-5595c9f964869b5fba2aa01375ba4c95531acaed89dd8061ba02ec7b6a28544d WatchSource:0}: Error finding container 5595c9f964869b5fba2aa01375ba4c95531acaed89dd8061ba02ec7b6a28544d: Status 404 returned error can't find the container with id 5595c9f964869b5fba2aa01375ba4c95531acaed89dd8061ba02ec7b6a28544d Jul 15 12:18:57.384811 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:57.384711 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-29vz4" event={"ID":"57ecd3c1-4b91-4730-b8da-9e55ae0204eb","Type":"ContainerStarted","Data":"291a694cbb675d76fa1e5c6b1d9da144e38f12cc9146c9936127c527c3252768"} Jul 15 12:18:57.384811 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:57.384755 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-29vz4" event={"ID":"57ecd3c1-4b91-4730-b8da-9e55ae0204eb","Type":"ContainerStarted","Data":"5595c9f964869b5fba2aa01375ba4c95531acaed89dd8061ba02ec7b6a28544d"} Jul 15 12:18:57.401835 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:18:57.401765 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-29vz4" podStartSLOduration=1.401752572 podStartE2EDuration="1.401752572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:18:57.399977053 +0000 UTC m=+461.034372000" watchObservedRunningTime="2026-07-15 12:18:57.401752572 +0000 UTC m=+461.036147519" Jul 15 12:19:08.361677 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:08.361635 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:08.418508 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:08.418471 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:11.252260 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:11.252218 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:11.367394 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:11.367362 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:11.936001 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:11.935955 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:12.052169 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.052135 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:12.174719 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.174678 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:12.249063 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.249033 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9"] Jul 15 12:19:12.252163 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.252144 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" Jul 15 12:19:12.255333 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.255313 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 12:19:12.255722 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.255586 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b7chz\"" Jul 15 12:19:12.255722 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.255594 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 12:19:12.255722 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.255639 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 12:19:12.255722 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.255654 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 12:19:12.270315 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.270296 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9"] Jul 15 12:19:12.347161 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.347134 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34d10c17-5d61-430f-9af9-10289dfd8171-tmp\") pod \"opendatahub-operator-controller-manager-85d97db8bc-zgqt9\" (UID: \"34d10c17-5d61-430f-9af9-10289dfd8171\") " pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" Jul 15 12:19:12.347161 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.347162 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/34d10c17-5d61-430f-9af9-10289dfd8171-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85d97db8bc-zgqt9\" (UID: \"34d10c17-5d61-430f-9af9-10289dfd8171\") " pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" Jul 15 12:19:12.347319 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.347212 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34d10c17-5d61-430f-9af9-10289dfd8171-webhook-cert\") pod \"opendatahub-operator-controller-manager-85d97db8bc-zgqt9\" (UID: \"34d10c17-5d61-430f-9af9-10289dfd8171\") " pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" Jul 15 12:19:12.347319 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.347228 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j922d\" (UniqueName: \"kubernetes.io/projected/34d10c17-5d61-430f-9af9-10289dfd8171-kube-api-access-j922d\") pod \"opendatahub-operator-controller-manager-85d97db8bc-zgqt9\" (UID: \"34d10c17-5d61-430f-9af9-10289dfd8171\") " pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" Jul 15 12:19:12.347319 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.347248 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/34d10c17-5d61-430f-9af9-10289dfd8171-manifests\") pod \"opendatahub-operator-controller-manager-85d97db8bc-zgqt9\" (UID: \"34d10c17-5d61-430f-9af9-10289dfd8171\") " pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" Jul 15 12:19:12.384142 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.384111 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:12.447698 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.447678 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34d10c17-5d61-430f-9af9-10289dfd8171-webhook-cert\") pod \"opendatahub-operator-controller-manager-85d97db8bc-zgqt9\" (UID: \"34d10c17-5d61-430f-9af9-10289dfd8171\") " pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" Jul 15 12:19:12.447854 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.447702 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j922d\" (UniqueName: \"kubernetes.io/projected/34d10c17-5d61-430f-9af9-10289dfd8171-kube-api-access-j922d\") pod \"opendatahub-operator-controller-manager-85d97db8bc-zgqt9\" (UID: \"34d10c17-5d61-430f-9af9-10289dfd8171\") " pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" Jul 15 12:19:12.447854 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.447724 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/34d10c17-5d61-430f-9af9-10289dfd8171-manifests\") pod \"opendatahub-operator-controller-manager-85d97db8bc-zgqt9\" (UID: \"34d10c17-5d61-430f-9af9-10289dfd8171\") " pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" Jul 15 12:19:12.447854 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.447746 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34d10c17-5d61-430f-9af9-10289dfd8171-tmp\") pod \"opendatahub-operator-controller-manager-85d97db8bc-zgqt9\" (UID: \"34d10c17-5d61-430f-9af9-10289dfd8171\") " pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" Jul 15 12:19:12.447854 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.447763 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/34d10c17-5d61-430f-9af9-10289dfd8171-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85d97db8bc-zgqt9\" (UID: \"34d10c17-5d61-430f-9af9-10289dfd8171\") " pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" Jul 15 12:19:12.448197 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.448177 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/34d10c17-5d61-430f-9af9-10289dfd8171-manifests\") pod \"opendatahub-operator-controller-manager-85d97db8bc-zgqt9\" (UID: \"34d10c17-5d61-430f-9af9-10289dfd8171\") " pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" Jul 15 12:19:12.450194 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.450163 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34d10c17-5d61-430f-9af9-10289dfd8171-tmp\") pod \"opendatahub-operator-controller-manager-85d97db8bc-zgqt9\" (UID: \"34d10c17-5d61-430f-9af9-10289dfd8171\") " pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" Jul 15 12:19:12.450271 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.450251 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34d10c17-5d61-430f-9af9-10289dfd8171-webhook-cert\") pod \"opendatahub-operator-controller-manager-85d97db8bc-zgqt9\" (UID: \"34d10c17-5d61-430f-9af9-10289dfd8171\") " pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" Jul 15 12:19:12.450318 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.450303 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/34d10c17-5d61-430f-9af9-10289dfd8171-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85d97db8bc-zgqt9\" (UID: \"34d10c17-5d61-430f-9af9-10289dfd8171\") " pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" Jul 15 12:19:12.457057 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.457036 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j922d\" (UniqueName: \"kubernetes.io/projected/34d10c17-5d61-430f-9af9-10289dfd8171-kube-api-access-j922d\") pod \"opendatahub-operator-controller-manager-85d97db8bc-zgqt9\" (UID: \"34d10c17-5d61-430f-9af9-10289dfd8171\") " pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" Jul 15 12:19:12.483862 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.483837 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:12.561735 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.561665 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" Jul 15 12:19:12.585863 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.585828 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:12.679131 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.679080 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:12.693140 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.693104 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9"] Jul 15 12:19:12.696362 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:19:12.696339 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d10c17_5d61_430f_9af9_10289dfd8171.slice/crio-7e6844c2f50be5e169e8bd919df81daaf19608ed9949d8460795e95caf7ae682 WatchSource:0}: Error finding container 7e6844c2f50be5e169e8bd919df81daaf19608ed9949d8460795e95caf7ae682: Status 404 returned error can't find the container with id 7e6844c2f50be5e169e8bd919df81daaf19608ed9949d8460795e95caf7ae682 Jul 15 12:19:12.742022 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.741990 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:12.854282 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:12.854189 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:13.445499 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:13.445421 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" event={"ID":"34d10c17-5d61-430f-9af9-10289dfd8171","Type":"ContainerStarted","Data":"7e6844c2f50be5e169e8bd919df81daaf19608ed9949d8460795e95caf7ae682"} Jul 15 12:19:16.459582 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:16.459547 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" event={"ID":"34d10c17-5d61-430f-9af9-10289dfd8171","Type":"ContainerStarted","Data":"950b3ee216d69369c52839e3ebd0cc08fd198c18a7c1f4952cb79814c2d8f9d5"} Jul 15 12:19:17.464723 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:17.464683 2556 generic.go:356] "Generic (PLEG): container finished" podID="34d10c17-5d61-430f-9af9-10289dfd8171" containerID="950b3ee216d69369c52839e3ebd0cc08fd198c18a7c1f4952cb79814c2d8f9d5" exitCode=0 Jul 15 12:19:17.464723 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:17.464724 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" event={"ID":"34d10c17-5d61-430f-9af9-10289dfd8171","Type":"ContainerDied","Data":"950b3ee216d69369c52839e3ebd0cc08fd198c18a7c1f4952cb79814c2d8f9d5"} Jul 15 12:19:18.471602 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:18.471567 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" event={"ID":"34d10c17-5d61-430f-9af9-10289dfd8171","Type":"ContainerStarted","Data":"a56a7b9551cf45b6bcecfc1708b76b48b226c7bc6ab858ff8fc87457e69ca41f"} Jul 15 12:19:18.472100 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:18.471680 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" Jul 15 12:19:18.495232 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:18.495184 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" podStartSLOduration=1.923807582 podStartE2EDuration="6.495165263s" totalImagesPullingTime="3.56168315s" totalInitContainerRuntime="1.009674531s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:19:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:19:18.493411903 +0000 UTC m=+482.127806851" watchObservedRunningTime="2026-07-15 12:19:18.495165263 +0000 UTC m=+482.129560270" Jul 15 12:19:22.868348 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:22.868316 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:22.969653 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:22.969614 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:23.782203 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:23.782162 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:23.882431 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:23.882397 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:29.479611 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:29.479583 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-85d97db8bc-zgqt9" Jul 15 12:19:29.484895 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:29.484857 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:31.166464 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:31.166431 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:31.354619 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:31.354578 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:32.019288 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:32.019250 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:32.120509 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:32.120467 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:32.228328 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:32.228290 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:32.424922 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:32.424835 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:32.624187 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:32.624158 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:32.954660 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:32.954620 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:33.385996 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:33.385964 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:35.434957 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:35.434921 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:40.886425 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:40.886392 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:40.943356 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:40.943324 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jwf49"] Jul 15 12:19:40.946496 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:40.946471 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jwf49" Jul 15 12:19:40.949442 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:40.949421 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 15 12:19:40.949550 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:40.949498 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-smwvg\"" Jul 15 12:19:40.952905 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:40.952721 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tm7pl"] Jul 15 12:19:40.955590 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:40.955572 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tm7pl" Jul 15 12:19:40.959113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:40.959096 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 15 12:19:40.959301 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:40.959282 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-b9r9s\"" Jul 15 12:19:40.959380 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:40.959365 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jwf49"] Jul 15 12:19:40.968891 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:40.968638 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tm7pl"] Jul 15 12:19:40.978702 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:40.978674 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:41.047365 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:41.047333 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e03cfc96-db62-4148-b246-53dc2077df25-cert\") pod \"kserve-controller-manager-d6f4b65b8-jwf49\" (UID: \"e03cfc96-db62-4148-b246-53dc2077df25\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jwf49" Jul 15 12:19:41.047365 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:41.047364 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhfsw\" (UniqueName: \"kubernetes.io/projected/e03cfc96-db62-4148-b246-53dc2077df25-kube-api-access-mhfsw\") pod \"kserve-controller-manager-d6f4b65b8-jwf49\" (UID: \"e03cfc96-db62-4148-b246-53dc2077df25\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jwf49" Jul 15 12:19:41.047539 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:41.047408 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d92b1bf-91bf-4159-8c51-1f8cb532d7b7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tm7pl\" (UID: \"1d92b1bf-91bf-4159-8c51-1f8cb532d7b7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tm7pl" Jul 15 12:19:41.047539 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:41.047447 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc2kx\" (UniqueName: \"kubernetes.io/projected/1d92b1bf-91bf-4159-8c51-1f8cb532d7b7-kube-api-access-mc2kx\") pod \"llmisvc-controller-manager-777c6bd7d5-tm7pl\" (UID: \"1d92b1bf-91bf-4159-8c51-1f8cb532d7b7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tm7pl" Jul 15 12:19:41.147983 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:41.147890 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc2kx\" (UniqueName: \"kubernetes.io/projected/1d92b1bf-91bf-4159-8c51-1f8cb532d7b7-kube-api-access-mc2kx\") pod \"llmisvc-controller-manager-777c6bd7d5-tm7pl\" (UID: \"1d92b1bf-91bf-4159-8c51-1f8cb532d7b7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tm7pl" Jul 15 12:19:41.147983 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:41.147941 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e03cfc96-db62-4148-b246-53dc2077df25-cert\") pod \"kserve-controller-manager-d6f4b65b8-jwf49\" (UID: \"e03cfc96-db62-4148-b246-53dc2077df25\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jwf49" Jul 15 12:19:41.147983 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:41.147964 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhfsw\" (UniqueName: \"kubernetes.io/projected/e03cfc96-db62-4148-b246-53dc2077df25-kube-api-access-mhfsw\") pod \"kserve-controller-manager-d6f4b65b8-jwf49\" (UID: \"e03cfc96-db62-4148-b246-53dc2077df25\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jwf49" Jul 15 12:19:41.148244 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:19:41.148066 2556 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 15 12:19:41.148244 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:19:41.148124 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e03cfc96-db62-4148-b246-53dc2077df25-cert podName:e03cfc96-db62-4148-b246-53dc2077df25 nodeName:}" failed. No retries permitted until 2026-07-15 12:19:41.648107875 +0000 UTC m=+505.282502801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e03cfc96-db62-4148-b246-53dc2077df25-cert") pod "kserve-controller-manager-d6f4b65b8-jwf49" (UID: "e03cfc96-db62-4148-b246-53dc2077df25") : secret "kserve-webhook-server-cert" not found Jul 15 12:19:41.148244 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:41.148150 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d92b1bf-91bf-4159-8c51-1f8cb532d7b7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tm7pl\" (UID: \"1d92b1bf-91bf-4159-8c51-1f8cb532d7b7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tm7pl" Jul 15 12:19:41.148398 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:19:41.148287 2556 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 15 12:19:41.148398 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:19:41.148355 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d92b1bf-91bf-4159-8c51-1f8cb532d7b7-cert podName:1d92b1bf-91bf-4159-8c51-1f8cb532d7b7 nodeName:}" failed. No retries permitted until 2026-07-15 12:19:41.64834139 +0000 UTC m=+505.282736317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1d92b1bf-91bf-4159-8c51-1f8cb532d7b7-cert") pod "llmisvc-controller-manager-777c6bd7d5-tm7pl" (UID: "1d92b1bf-91bf-4159-8c51-1f8cb532d7b7") : secret "llmisvc-webhook-server-cert" not found Jul 15 12:19:41.168955 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:41.168922 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc2kx\" (UniqueName: \"kubernetes.io/projected/1d92b1bf-91bf-4159-8c51-1f8cb532d7b7-kube-api-access-mc2kx\") pod \"llmisvc-controller-manager-777c6bd7d5-tm7pl\" (UID: \"1d92b1bf-91bf-4159-8c51-1f8cb532d7b7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tm7pl" Jul 15 12:19:41.179344 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:41.179314 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhfsw\" (UniqueName: \"kubernetes.io/projected/e03cfc96-db62-4148-b246-53dc2077df25-kube-api-access-mhfsw\") pod \"kserve-controller-manager-d6f4b65b8-jwf49\" (UID: \"e03cfc96-db62-4148-b246-53dc2077df25\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jwf49" Jul 15 12:19:41.291395 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:41.291358 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:41.430574 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:41.430477 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:41.652551 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:41.652521 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e03cfc96-db62-4148-b246-53dc2077df25-cert\") pod \"kserve-controller-manager-d6f4b65b8-jwf49\" (UID: \"e03cfc96-db62-4148-b246-53dc2077df25\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jwf49" Jul 15 12:19:41.652742 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:41.652580 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d92b1bf-91bf-4159-8c51-1f8cb532d7b7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tm7pl\" (UID: \"1d92b1bf-91bf-4159-8c51-1f8cb532d7b7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tm7pl" Jul 15 12:19:41.655138 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:41.655111 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d92b1bf-91bf-4159-8c51-1f8cb532d7b7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tm7pl\" (UID: \"1d92b1bf-91bf-4159-8c51-1f8cb532d7b7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tm7pl" Jul 15 12:19:41.655246 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:41.655111 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e03cfc96-db62-4148-b246-53dc2077df25-cert\") pod \"kserve-controller-manager-d6f4b65b8-jwf49\" (UID: \"e03cfc96-db62-4148-b246-53dc2077df25\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jwf49" Jul 15 12:19:41.856813 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:41.856760 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jwf49" Jul 15 12:19:41.869317 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:41.867615 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tm7pl" Jul 15 12:19:42.004047 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.003931 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jwf49"] Jul 15 12:19:42.006542 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:19:42.006508 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03cfc96_db62_4148_b246_53dc2077df25.slice/crio-19a2dedd0edaa7c14390bf9407fb0dee5528c48924c3dec8e58243c2926bcd64 WatchSource:0}: Error finding container 19a2dedd0edaa7c14390bf9407fb0dee5528c48924c3dec8e58243c2926bcd64: Status 404 returned error can't find the container with id 19a2dedd0edaa7c14390bf9407fb0dee5528c48924c3dec8e58243c2926bcd64 Jul 15 12:19:42.024159 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.024136 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tm7pl"] Jul 15 12:19:42.027022 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:19:42.027000 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1d92b1bf_91bf_4159_8c51_1f8cb532d7b7.slice/crio-95ac2f39a71e532dda56a24639a7a82a7b9e37b0d519313b4cc6bf887ea3789c WatchSource:0}: Error finding container 95ac2f39a71e532dda56a24639a7a82a7b9e37b0d519313b4cc6bf887ea3789c: Status 404 returned error can't find the container with id 95ac2f39a71e532dda56a24639a7a82a7b9e37b0d519313b4cc6bf887ea3789c Jul 15 12:19:42.235457 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.235370 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:42.347293 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.347257 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6795b9c646-8tt8d"] Jul 15 12:19:42.351685 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.351663 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6795b9c646-8tt8d" Jul 15 12:19:42.356743 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.356725 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 12:19:42.357242 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.357225 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 12:19:42.369648 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.369627 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 12:19:42.369722 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.369682 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7bc6d\"" Jul 15 12:19:42.369913 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.369900 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 12:19:42.385018 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.384995 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6795b9c646-8tt8d"] Jul 15 12:19:42.401345 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.401314 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:42.457402 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.457378 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59m6l\" (UniqueName: \"kubernetes.io/projected/7cf9010d-bdc5-4a92-8671-cb5b3e7497f2-kube-api-access-59m6l\") pod \"kube-auth-proxy-6795b9c646-8tt8d\" (UID: \"7cf9010d-bdc5-4a92-8671-cb5b3e7497f2\") " pod="openshift-ingress/kube-auth-proxy-6795b9c646-8tt8d" Jul 15 12:19:42.457537 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.457420 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cf9010d-bdc5-4a92-8671-cb5b3e7497f2-tmp\") pod \"kube-auth-proxy-6795b9c646-8tt8d\" (UID: \"7cf9010d-bdc5-4a92-8671-cb5b3e7497f2\") " pod="openshift-ingress/kube-auth-proxy-6795b9c646-8tt8d" Jul 15 12:19:42.457537 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.457486 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf9010d-bdc5-4a92-8671-cb5b3e7497f2-tls-certs\") pod \"kube-auth-proxy-6795b9c646-8tt8d\" (UID: \"7cf9010d-bdc5-4a92-8671-cb5b3e7497f2\") " pod="openshift-ingress/kube-auth-proxy-6795b9c646-8tt8d" Jul 15 12:19:42.485240 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.485208 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:42.554700 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.554671 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tm7pl" event={"ID":"1d92b1bf-91bf-4159-8c51-1f8cb532d7b7","Type":"ContainerStarted","Data":"95ac2f39a71e532dda56a24639a7a82a7b9e37b0d519313b4cc6bf887ea3789c"} Jul 15 12:19:42.555747 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.555720 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jwf49" event={"ID":"e03cfc96-db62-4148-b246-53dc2077df25","Type":"ContainerStarted","Data":"19a2dedd0edaa7c14390bf9407fb0dee5528c48924c3dec8e58243c2926bcd64"} Jul 15 12:19:42.558090 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.558074 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf9010d-bdc5-4a92-8671-cb5b3e7497f2-tls-certs\") pod \"kube-auth-proxy-6795b9c646-8tt8d\" (UID: \"7cf9010d-bdc5-4a92-8671-cb5b3e7497f2\") " pod="openshift-ingress/kube-auth-proxy-6795b9c646-8tt8d" Jul 15 12:19:42.558161 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.558104 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59m6l\" (UniqueName: \"kubernetes.io/projected/7cf9010d-bdc5-4a92-8671-cb5b3e7497f2-kube-api-access-59m6l\") pod \"kube-auth-proxy-6795b9c646-8tt8d\" (UID: \"7cf9010d-bdc5-4a92-8671-cb5b3e7497f2\") " pod="openshift-ingress/kube-auth-proxy-6795b9c646-8tt8d" Jul 15 12:19:42.558257 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.558240 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cf9010d-bdc5-4a92-8671-cb5b3e7497f2-tmp\") pod \"kube-auth-proxy-6795b9c646-8tt8d\" (UID: \"7cf9010d-bdc5-4a92-8671-cb5b3e7497f2\") " pod="openshift-ingress/kube-auth-proxy-6795b9c646-8tt8d" Jul 15 12:19:42.560459 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.560439 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cf9010d-bdc5-4a92-8671-cb5b3e7497f2-tmp\") pod \"kube-auth-proxy-6795b9c646-8tt8d\" (UID: \"7cf9010d-bdc5-4a92-8671-cb5b3e7497f2\") " pod="openshift-ingress/kube-auth-proxy-6795b9c646-8tt8d" Jul 15 12:19:42.560584 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.560569 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf9010d-bdc5-4a92-8671-cb5b3e7497f2-tls-certs\") pod \"kube-auth-proxy-6795b9c646-8tt8d\" (UID: \"7cf9010d-bdc5-4a92-8671-cb5b3e7497f2\") " pod="openshift-ingress/kube-auth-proxy-6795b9c646-8tt8d" Jul 15 12:19:42.567668 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.567644 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59m6l\" (UniqueName: \"kubernetes.io/projected/7cf9010d-bdc5-4a92-8671-cb5b3e7497f2-kube-api-access-59m6l\") pod \"kube-auth-proxy-6795b9c646-8tt8d\" (UID: \"7cf9010d-bdc5-4a92-8671-cb5b3e7497f2\") " pod="openshift-ingress/kube-auth-proxy-6795b9c646-8tt8d" Jul 15 12:19:42.660855 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.660825 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6795b9c646-8tt8d" Jul 15 12:19:42.791600 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:42.791568 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6795b9c646-8tt8d"] Jul 15 12:19:42.794235 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:19:42.794203 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf9010d_bdc5_4a92_8671_cb5b3e7497f2.slice/crio-0f92903bcbac84bd8741409a5573c75ef18538ffd3c56a1e6551f51408773992 WatchSource:0}: Error finding container 0f92903bcbac84bd8741409a5573c75ef18538ffd3c56a1e6551f51408773992: Status 404 returned error can't find the container with id 0f92903bcbac84bd8741409a5573c75ef18538ffd3c56a1e6551f51408773992 Jul 15 12:19:43.561739 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:43.561695 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6795b9c646-8tt8d" event={"ID":"7cf9010d-bdc5-4a92-8671-cb5b3e7497f2","Type":"ContainerStarted","Data":"0f92903bcbac84bd8741409a5573c75ef18538ffd3c56a1e6551f51408773992"} Jul 15 12:19:43.956828 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:43.956697 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq"] Jul 15 12:19:43.960102 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:43.960076 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:43.963177 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:43.963153 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-78tww\"" Jul 15 12:19:43.963414 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:43.963395 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 15 12:19:43.963609 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:43.963591 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 12:19:43.963921 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:43.963903 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 15 12:19:43.964191 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:43.964172 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 15 12:19:43.974498 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:43.973424 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq"] Jul 15 12:19:43.989090 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:43.989062 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:44.060602 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.060567 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:44.076599 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.076564 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.076769 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.076649 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.076769 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.076701 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv6q9\" (UniqueName: \"kubernetes.io/projected/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-kube-api-access-zv6q9\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.076769 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.076730 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.076769 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.076757 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.077004 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.076798 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.077004 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.076860 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.178126 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.178071 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.178126 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.178127 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.178411 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.178173 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.178411 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.178222 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.178411 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.178262 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv6q9\" (UniqueName: \"kubernetes.io/projected/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-kube-api-access-zv6q9\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.178411 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.178289 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.178411 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.178314 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.179471 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.179441 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.182285 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.182238 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.183072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.183026 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.183157 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.183143 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.183260 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.183241 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.189864 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.189839 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.190198 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.190155 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv6q9\" (UniqueName: \"kubernetes.io/projected/9ed8cc67-c7ae-4c2e-8b81-df7808745d1a-kube-api-access-zv6q9\") pod \"istiod-openshift-gateway-94bb8fbfd-blmkq\" (UID: \"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.275042 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.275010 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:44.854947 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:44.854924 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq"] Jul 15 12:19:44.855277 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:19:44.855041 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed8cc67_c7ae_4c2e_8b81_df7808745d1a.slice/crio-2906e7eded6a56732bedd79bdd43c2bf06362cbe8e36a3adf1ad9565d657b08c WatchSource:0}: Error finding container 2906e7eded6a56732bedd79bdd43c2bf06362cbe8e36a3adf1ad9565d657b08c: Status 404 returned error can't find the container with id 2906e7eded6a56732bedd79bdd43c2bf06362cbe8e36a3adf1ad9565d657b08c Jul 15 12:19:45.163791 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:45.163738 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:45.573744 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:45.573707 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jwf49" event={"ID":"e03cfc96-db62-4148-b246-53dc2077df25","Type":"ContainerStarted","Data":"ac3e2d6272875fa8039819ed22ba6eaaf2d7ad7913572e9711aefe509227bc7d"} Jul 15 12:19:45.573945 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:45.573810 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jwf49" Jul 15 12:19:45.575031 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:45.575002 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" event={"ID":"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a","Type":"ContainerStarted","Data":"2906e7eded6a56732bedd79bdd43c2bf06362cbe8e36a3adf1ad9565d657b08c"} Jul 15 12:19:45.592000 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:45.591956 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jwf49" podStartSLOduration=2.861640197 podStartE2EDuration="5.591936989s" totalImagesPullingTime="2.730296792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:19:45.59036478 +0000 UTC m=+509.224759728" watchObservedRunningTime="2026-07-15 12:19:45.591936989 +0000 UTC m=+509.226331940" Jul 15 12:19:46.582641 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:46.582598 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tm7pl" event={"ID":"1d92b1bf-91bf-4159-8c51-1f8cb532d7b7","Type":"ContainerStarted","Data":"2384eb6a586104d4a5c48871812568a2c8490eab299e462df0778091246e98c3"} Jul 15 12:19:46.583147 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:46.582718 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tm7pl" Jul 15 12:19:46.584424 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:46.584394 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6795b9c646-8tt8d" event={"ID":"7cf9010d-bdc5-4a92-8671-cb5b3e7497f2","Type":"ContainerStarted","Data":"52e9ec5cfde30cea9f718b21ace9586c2d51e97c7d037c0e67c6e6387c1af015"} Jul 15 12:19:46.600689 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:46.600634 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tm7pl" podStartSLOduration=2.408291348 podStartE2EDuration="6.600619494s" totalImagesPullingTime="4.192328146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:19:46.599546836 +0000 UTC m=+510.233941785" watchObservedRunningTime="2026-07-15 12:19:46.600619494 +0000 UTC m=+510.235014442" Jul 15 12:19:46.616754 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:46.616700 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6795b9c646-8tt8d" podStartSLOduration=1.189364879 podStartE2EDuration="4.616680006s" totalImagesPullingTime="3.427315127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:19:46.615705528 +0000 UTC m=+510.250100476" watchObservedRunningTime="2026-07-15 12:19:46.616680006 +0000 UTC m=+510.251074955" Jul 15 12:19:48.094859 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:48.094816 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:19:48.095247 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:48.094913 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:19:48.594280 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:48.594247 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" event={"ID":"9ed8cc67-c7ae-4c2e-8b81-df7808745d1a","Type":"ContainerStarted","Data":"e4b3afc151699654814f90c2f5201f5731635e6424b767ea5d11094b5ae99b65"} Jul 15 12:19:48.594458 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:48.594359 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:48.621028 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:48.620976 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" podStartSLOduration=2.383708785 podStartE2EDuration="5.620959622s" totalImagesPullingTime="3.237250837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:19:48.620823613 +0000 UTC m=+512.255218563" watchObservedRunningTime="2026-07-15 12:19:48.620959622 +0000 UTC m=+512.255354570" Jul 15 12:19:48.670495 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:48.670446 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:48.773456 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:48.773417 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:48.954971 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:48.954899 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:49.210824 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:49.210702 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:49.516469 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:49.516439 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:49.600487 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:49.600463 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-blmkq" Jul 15 12:19:54.378975 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:54.378943 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:54.479100 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:19:54.479061 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:20:16.589517 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:20:16.589487 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jwf49" Jul 15 12:20:17.591077 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:20:17.591044 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tm7pl" Jul 15 12:20:21.510977 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:20:21.510940 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:20:32.822237 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:20:32.822197 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:20:32.921781 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:20:32.921748 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:20:33.021888 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:20:33.021852 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:20:33.768901 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:20:33.768868 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:20:33.971506 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:20:33.971472 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:20:58.534299 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:20:58.534253 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:02.136166 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:02.136124 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:05.159426 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:05.159393 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:06.646844 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:06.646804 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:06.745799 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:06.745749 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:06.847827 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:06.847788 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:07.297718 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:07.297686 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:07.402217 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:07.402184 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:09.342409 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:09.342375 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc"] Jul 15 12:21:09.344431 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:09.344412 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc" Jul 15 12:21:09.347244 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:09.347224 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 12:21:09.348604 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:09.348581 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-pq2bh\"" Jul 15 12:21:09.348690 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:09.348613 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 12:21:09.354043 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:09.354021 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc"] Jul 15 12:21:09.402800 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:09.402744 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbcgs\" (UniqueName: \"kubernetes.io/projected/99c1a2de-fcf1-417c-b77f-6d543b4f26ee-kube-api-access-hbcgs\") pod \"limitador-operator-controller-manager-859d7b78bd-rswbc\" (UID: \"99c1a2de-fcf1-417c-b77f-6d543b4f26ee\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc" Jul 15 12:21:09.503536 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:09.503497 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbcgs\" (UniqueName: \"kubernetes.io/projected/99c1a2de-fcf1-417c-b77f-6d543b4f26ee-kube-api-access-hbcgs\") pod \"limitador-operator-controller-manager-859d7b78bd-rswbc\" (UID: \"99c1a2de-fcf1-417c-b77f-6d543b4f26ee\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc" Jul 15 12:21:09.511251 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:09.511221 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbcgs\" (UniqueName: \"kubernetes.io/projected/99c1a2de-fcf1-417c-b77f-6d543b4f26ee-kube-api-access-hbcgs\") pod \"limitador-operator-controller-manager-859d7b78bd-rswbc\" (UID: \"99c1a2de-fcf1-417c-b77f-6d543b4f26ee\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc" Jul 15 12:21:09.654194 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:09.654114 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc" Jul 15 12:21:09.730810 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:09.730760 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:09.788221 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:09.788197 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc"] Jul 15 12:21:09.790444 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:21:09.790424 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c1a2de_fcf1_417c_b77f_6d543b4f26ee.slice/crio-75bb7e2dd94c2b7338eb7d9e469552a2cc79f123a4b5ad8af2344ddfda94650f WatchSource:0}: Error finding container 75bb7e2dd94c2b7338eb7d9e469552a2cc79f123a4b5ad8af2344ddfda94650f: Status 404 returned error can't find the container with id 75bb7e2dd94c2b7338eb7d9e469552a2cc79f123a4b5ad8af2344ddfda94650f Jul 15 12:21:09.885465 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:09.885431 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc" event={"ID":"99c1a2de-fcf1-417c-b77f-6d543b4f26ee","Type":"ContainerStarted","Data":"75bb7e2dd94c2b7338eb7d9e469552a2cc79f123a4b5ad8af2344ddfda94650f"} Jul 15 12:21:10.149423 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:10.149387 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:10.384871 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:10.384828 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:11.896586 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:11.896480 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc" event={"ID":"99c1a2de-fcf1-417c-b77f-6d543b4f26ee","Type":"ContainerStarted","Data":"77929ecc07930b17e4275b5bdee42facc04adb59d18810f1c2beb2c349e6e44b"} Jul 15 12:21:11.897019 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:11.896609 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc" Jul 15 12:21:11.912248 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:11.912201 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc" podStartSLOduration=1.073839679 podStartE2EDuration="2.912188403s" totalImagesPullingTime="1.838348724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:21:11.911271334 +0000 UTC m=+595.545666282" watchObservedRunningTime="2026-07-15 12:21:11.912188403 +0000 UTC m=+595.546583413" Jul 15 12:21:13.163761 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:13.163734 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:14.584720 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:14.584687 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:14.684981 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:14.684942 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:14.862003 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:14.861915 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:14.985562 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:14.985528 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:16.724674 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:16.724639 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:21:16.725180 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:16.724682 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:21:17.355899 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:17.355862 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:17.456058 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:17.456026 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:17.557329 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:17.557295 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:18.333886 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:18.333846 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:18.390555 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:18.390522 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:22.902308 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:22.902280 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc" Jul 15 12:21:25.223707 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:25.223676 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ksz9k"] Jul 15 12:21:25.225708 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:25.225693 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ksz9k" Jul 15 12:21:25.228911 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:25.228884 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jps4v\"" Jul 15 12:21:25.239487 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:25.239466 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ksz9k"] Jul 15 12:21:25.331923 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:25.331890 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84dee9c1-a838-48ea-b30c-00ad44c2ad3d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ksz9k\" (UID: \"84dee9c1-a838-48ea-b30c-00ad44c2ad3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ksz9k" Jul 15 12:21:25.331923 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:25.331929 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9jlv\" (UniqueName: \"kubernetes.io/projected/84dee9c1-a838-48ea-b30c-00ad44c2ad3d-kube-api-access-b9jlv\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ksz9k\" (UID: \"84dee9c1-a838-48ea-b30c-00ad44c2ad3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ksz9k" Jul 15 12:21:25.433304 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:25.433273 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84dee9c1-a838-48ea-b30c-00ad44c2ad3d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ksz9k\" (UID: \"84dee9c1-a838-48ea-b30c-00ad44c2ad3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ksz9k" Jul 15 12:21:25.433450 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:25.433311 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9jlv\" (UniqueName: \"kubernetes.io/projected/84dee9c1-a838-48ea-b30c-00ad44c2ad3d-kube-api-access-b9jlv\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ksz9k\" (UID: \"84dee9c1-a838-48ea-b30c-00ad44c2ad3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ksz9k" Jul 15 12:21:25.433652 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:25.433634 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84dee9c1-a838-48ea-b30c-00ad44c2ad3d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ksz9k\" (UID: \"84dee9c1-a838-48ea-b30c-00ad44c2ad3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ksz9k" Jul 15 12:21:25.442597 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:25.442565 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9jlv\" (UniqueName: \"kubernetes.io/projected/84dee9c1-a838-48ea-b30c-00ad44c2ad3d-kube-api-access-b9jlv\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ksz9k\" (UID: \"84dee9c1-a838-48ea-b30c-00ad44c2ad3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ksz9k" Jul 15 12:21:25.535285 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:25.535262 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ksz9k" Jul 15 12:21:25.669069 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:25.669046 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ksz9k"] Jul 15 12:21:25.671827 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:21:25.671797 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84dee9c1_a838_48ea_b30c_00ad44c2ad3d.slice/crio-f3f50b52aeddd5522e4f852453964ae16e0e3d5b121de1f715f51104bb5545fb WatchSource:0}: Error finding container f3f50b52aeddd5522e4f852453964ae16e0e3d5b121de1f715f51104bb5545fb: Status 404 returned error can't find the container with id f3f50b52aeddd5522e4f852453964ae16e0e3d5b121de1f715f51104bb5545fb Jul 15 12:21:25.774838 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:25.774809 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:25.879888 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:25.879808 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:25.954572 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:25.954537 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ksz9k" event={"ID":"84dee9c1-a838-48ea-b30c-00ad44c2ad3d","Type":"ContainerStarted","Data":"f3f50b52aeddd5522e4f852453964ae16e0e3d5b121de1f715f51104bb5545fb"} Jul 15 12:21:26.023927 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.023893 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ksz9k"] Jul 15 12:21:26.028581 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.028557 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ksz9k"] Jul 15 12:21:26.061025 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.060985 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc"] Jul 15 12:21:26.061273 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.061240 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc" podUID="99c1a2de-fcf1-417c-b77f-6d543b4f26ee" containerName="manager" containerID="cri-o://77929ecc07930b17e4275b5bdee42facc04adb59d18810f1c2beb2c349e6e44b" gracePeriod=2 Jul 15 12:21:26.075128 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.075061 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc"] Jul 15 12:21:26.082976 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:21:26.082945 2556 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"99c1a2de-fcf1-417c-b77f-6d543b4f26ee\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-rswbc\": pods \"limitador-operator-controller-manager-859d7b78bd-rswbc\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc" Jul 15 12:21:26.089697 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.089672 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vqmb9"] Jul 15 12:21:26.089966 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.089954 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99c1a2de-fcf1-417c-b77f-6d543b4f26ee" containerName="manager" Jul 15 12:21:26.090012 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.089968 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="99c1a2de-fcf1-417c-b77f-6d543b4f26ee" containerName="manager" Jul 15 12:21:26.090046 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.090024 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="99c1a2de-fcf1-417c-b77f-6d543b4f26ee" containerName="manager" Jul 15 12:21:26.092144 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.092115 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vqmb9" Jul 15 12:21:26.106753 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.106729 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vqmb9"] Jul 15 12:21:26.137526 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:21:26.137459 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rswbc\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="99c1a2de-fcf1-417c-b77f-6d543b4f26ee" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc" Jul 15 12:21:26.241112 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.241041 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9qp6\" (UniqueName: \"kubernetes.io/projected/681b92ea-b948-41b1-ab29-e6e2044b5cdf-kube-api-access-c9qp6\") pod \"limitador-operator-controller-manager-859d7b78bd-vqmb9\" (UID: \"681b92ea-b948-41b1-ab29-e6e2044b5cdf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vqmb9" Jul 15 12:21:26.318052 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.318021 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc" Jul 15 12:21:26.320637 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:21:26.320605 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rswbc\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="99c1a2de-fcf1-417c-b77f-6d543b4f26ee" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc" Jul 15 12:21:26.341754 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.341723 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9qp6\" (UniqueName: \"kubernetes.io/projected/681b92ea-b948-41b1-ab29-e6e2044b5cdf-kube-api-access-c9qp6\") pod \"limitador-operator-controller-manager-859d7b78bd-vqmb9\" (UID: \"681b92ea-b948-41b1-ab29-e6e2044b5cdf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vqmb9" Jul 15 12:21:26.356391 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.356354 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9qp6\" (UniqueName: \"kubernetes.io/projected/681b92ea-b948-41b1-ab29-e6e2044b5cdf-kube-api-access-c9qp6\") pod \"limitador-operator-controller-manager-859d7b78bd-vqmb9\" (UID: \"681b92ea-b948-41b1-ab29-e6e2044b5cdf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vqmb9" Jul 15 12:21:26.442314 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.442239 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vqmb9" Jul 15 12:21:26.442314 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.442287 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbcgs\" (UniqueName: \"kubernetes.io/projected/99c1a2de-fcf1-417c-b77f-6d543b4f26ee-kube-api-access-hbcgs\") pod \"99c1a2de-fcf1-417c-b77f-6d543b4f26ee\" (UID: \"99c1a2de-fcf1-417c-b77f-6d543b4f26ee\") " Jul 15 12:21:26.445129 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.445096 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c1a2de-fcf1-417c-b77f-6d543b4f26ee-kube-api-access-hbcgs" (OuterVolumeSpecName: "kube-api-access-hbcgs") pod "99c1a2de-fcf1-417c-b77f-6d543b4f26ee" (UID: "99c1a2de-fcf1-417c-b77f-6d543b4f26ee"). InnerVolumeSpecName "kube-api-access-hbcgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:21:26.543348 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.543313 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbcgs\" (UniqueName: \"kubernetes.io/projected/99c1a2de-fcf1-417c-b77f-6d543b4f26ee-kube-api-access-hbcgs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:21:26.748835 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.748755 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vqmb9"] Jul 15 12:21:26.751017 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:21:26.750990 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681b92ea_b948_41b1_ab29_e6e2044b5cdf.slice/crio-7dc550bda417a98bab8dba62fcff6def7f5dbb829b2be703bd1e4a2765e2a671 WatchSource:0}: Error finding container 7dc550bda417a98bab8dba62fcff6def7f5dbb829b2be703bd1e4a2765e2a671: Status 404 returned error can't find the container with id 7dc550bda417a98bab8dba62fcff6def7f5dbb829b2be703bd1e4a2765e2a671 Jul 15 12:21:26.853026 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:21:26.852996 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rswbc\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="99c1a2de-fcf1-417c-b77f-6d543b4f26ee" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc" Jul 15 12:21:26.856519 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.856488 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99c1a2de-fcf1-417c-b77f-6d543b4f26ee" path="/var/lib/kubelet/pods/99c1a2de-fcf1-417c-b77f-6d543b4f26ee/volumes" Jul 15 12:21:26.960608 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.960520 2556 generic.go:356] "Generic (PLEG): container finished" podID="99c1a2de-fcf1-417c-b77f-6d543b4f26ee" containerID="77929ecc07930b17e4275b5bdee42facc04adb59d18810f1c2beb2c349e6e44b" exitCode=0 Jul 15 12:21:26.960608 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.960580 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rswbc" Jul 15 12:21:26.960832 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.960639 2556 scope.go:117] "RemoveContainer" containerID="77929ecc07930b17e4275b5bdee42facc04adb59d18810f1c2beb2c349e6e44b" Jul 15 12:21:26.962577 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.962550 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vqmb9" event={"ID":"681b92ea-b948-41b1-ab29-e6e2044b5cdf","Type":"ContainerStarted","Data":"e1a6c727199eac15b4b277a90ae1fda49ca1e0a0a89bca55bb5ec2c1e2fb66c4"} Jul 15 12:21:26.962703 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.962615 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vqmb9" event={"ID":"681b92ea-b948-41b1-ab29-e6e2044b5cdf","Type":"ContainerStarted","Data":"7dc550bda417a98bab8dba62fcff6def7f5dbb829b2be703bd1e4a2765e2a671"} Jul 15 12:21:26.962703 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.962674 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vqmb9" Jul 15 12:21:26.971631 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.971612 2556 scope.go:117] "RemoveContainer" containerID="77929ecc07930b17e4275b5bdee42facc04adb59d18810f1c2beb2c349e6e44b" Jul 15 12:21:26.971921 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:21:26.971898 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77929ecc07930b17e4275b5bdee42facc04adb59d18810f1c2beb2c349e6e44b\": container with ID starting with 77929ecc07930b17e4275b5bdee42facc04adb59d18810f1c2beb2c349e6e44b not found: ID does not exist" containerID="77929ecc07930b17e4275b5bdee42facc04adb59d18810f1c2beb2c349e6e44b" Jul 15 12:21:26.972025 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.971932 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77929ecc07930b17e4275b5bdee42facc04adb59d18810f1c2beb2c349e6e44b"} err="failed to get container status \"77929ecc07930b17e4275b5bdee42facc04adb59d18810f1c2beb2c349e6e44b\": rpc error: code = NotFound desc = could not find container \"77929ecc07930b17e4275b5bdee42facc04adb59d18810f1c2beb2c349e6e44b\": container with ID starting with 77929ecc07930b17e4275b5bdee42facc04adb59d18810f1c2beb2c349e6e44b not found: ID does not exist" Jul 15 12:21:26.988840 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:26.988794 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vqmb9" podStartSLOduration=0.988763536 podStartE2EDuration="988.763536ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:21:26.98750111 +0000 UTC m=+610.621896059" watchObservedRunningTime="2026-07-15 12:21:26.988763536 +0000 UTC m=+610.623158485" Jul 15 12:21:30.544587 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:30.544538 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:30.645367 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:30.645331 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:30.987301 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:30.987219 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ksz9k" podUID="84dee9c1-a838-48ea-b30c-00ad44c2ad3d" containerName="manager" containerID="cri-o://531469ebe7d10b78ed31cb96d0d42f1ba33f760a2e74537190160d2dfa4f82bd" gracePeriod=2 Jul 15 12:21:31.218334 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:31.218309 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ksz9k" Jul 15 12:21:31.220667 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:21:31.220641 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-ksz9k\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="84dee9c1-a838-48ea-b30c-00ad44c2ad3d" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ksz9k" Jul 15 12:21:31.382306 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:31.382268 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84dee9c1-a838-48ea-b30c-00ad44c2ad3d-extensions-socket-volume\") pod \"84dee9c1-a838-48ea-b30c-00ad44c2ad3d\" (UID: \"84dee9c1-a838-48ea-b30c-00ad44c2ad3d\") " Jul 15 12:21:31.382456 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:31.382320 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9jlv\" (UniqueName: \"kubernetes.io/projected/84dee9c1-a838-48ea-b30c-00ad44c2ad3d-kube-api-access-b9jlv\") pod \"84dee9c1-a838-48ea-b30c-00ad44c2ad3d\" (UID: \"84dee9c1-a838-48ea-b30c-00ad44c2ad3d\") " Jul 15 12:21:31.382561 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:31.382538 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84dee9c1-a838-48ea-b30c-00ad44c2ad3d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "84dee9c1-a838-48ea-b30c-00ad44c2ad3d" (UID: "84dee9c1-a838-48ea-b30c-00ad44c2ad3d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:21:31.384480 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:31.384464 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84dee9c1-a838-48ea-b30c-00ad44c2ad3d-kube-api-access-b9jlv" (OuterVolumeSpecName: "kube-api-access-b9jlv") pod "84dee9c1-a838-48ea-b30c-00ad44c2ad3d" (UID: "84dee9c1-a838-48ea-b30c-00ad44c2ad3d"). InnerVolumeSpecName "kube-api-access-b9jlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:21:31.482979 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:31.482950 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9jlv\" (UniqueName: \"kubernetes.io/projected/84dee9c1-a838-48ea-b30c-00ad44c2ad3d-kube-api-access-b9jlv\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:21:31.482979 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:31.482977 2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84dee9c1-a838-48ea-b30c-00ad44c2ad3d-extensions-socket-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:21:31.992640 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:31.992610 2556 generic.go:356] "Generic (PLEG): container finished" podID="84dee9c1-a838-48ea-b30c-00ad44c2ad3d" containerID="531469ebe7d10b78ed31cb96d0d42f1ba33f760a2e74537190160d2dfa4f82bd" exitCode=2 Jul 15 12:21:31.993093 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:31.992651 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ksz9k" Jul 15 12:21:31.993093 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:31.992664 2556 scope.go:117] "RemoveContainer" containerID="531469ebe7d10b78ed31cb96d0d42f1ba33f760a2e74537190160d2dfa4f82bd" Jul 15 12:21:31.995707 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:21:31.995681 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-ksz9k\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="84dee9c1-a838-48ea-b30c-00ad44c2ad3d" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ksz9k" Jul 15 12:21:32.003448 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:32.003433 2556 scope.go:117] "RemoveContainer" containerID="531469ebe7d10b78ed31cb96d0d42f1ba33f760a2e74537190160d2dfa4f82bd" Jul 15 12:21:32.003538 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:21:32.003512 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-ksz9k\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="84dee9c1-a838-48ea-b30c-00ad44c2ad3d" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ksz9k" Jul 15 12:21:32.003859 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:21:32.003836 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"531469ebe7d10b78ed31cb96d0d42f1ba33f760a2e74537190160d2dfa4f82bd\": container with ID starting with 531469ebe7d10b78ed31cb96d0d42f1ba33f760a2e74537190160d2dfa4f82bd not found: ID does not exist" containerID="531469ebe7d10b78ed31cb96d0d42f1ba33f760a2e74537190160d2dfa4f82bd" Jul 15 12:21:32.003912 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:32.003872 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531469ebe7d10b78ed31cb96d0d42f1ba33f760a2e74537190160d2dfa4f82bd"} err="failed to get container status \"531469ebe7d10b78ed31cb96d0d42f1ba33f760a2e74537190160d2dfa4f82bd\": rpc error: code = NotFound desc = could not find container \"531469ebe7d10b78ed31cb96d0d42f1ba33f760a2e74537190160d2dfa4f82bd\": container with ID starting with 531469ebe7d10b78ed31cb96d0d42f1ba33f760a2e74537190160d2dfa4f82bd not found: ID does not exist" Jul 15 12:21:32.854203 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:32.854169 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84dee9c1-a838-48ea-b30c-00ad44c2ad3d" path="/var/lib/kubelet/pods/84dee9c1-a838-48ea-b30c-00ad44c2ad3d/volumes" Jul 15 12:21:33.113812 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:33.113687 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:33.215907 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:33.215876 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:36.474164 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:36.474134 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:36.575577 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:36.575535 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:37.974902 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:37.974825 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vqmb9" Jul 15 12:21:43.506278 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:43.506229 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:43.608556 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:43.608521 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:48.304279 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:48.304246 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:48.402485 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:48.402454 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:50.611926 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:50.611891 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt"] Jul 15 12:21:50.612318 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:50.612155 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84dee9c1-a838-48ea-b30c-00ad44c2ad3d" containerName="manager" Jul 15 12:21:50.612318 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:50.612165 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="84dee9c1-a838-48ea-b30c-00ad44c2ad3d" containerName="manager" Jul 15 12:21:50.612318 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:50.612223 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="84dee9c1-a838-48ea-b30c-00ad44c2ad3d" containerName="manager" Jul 15 12:21:50.615398 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:50.615382 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt" Jul 15 12:21:50.618283 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:50.618265 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jps4v\"" Jul 15 12:21:50.627891 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:50.627872 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt"] Jul 15 12:21:50.634695 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:50.634675 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6rjq\" (UniqueName: \"kubernetes.io/projected/ebfea797-c25e-4207-8101-94aa78d9d543-kube-api-access-z6rjq\") pod \"kuadrant-operator-controller-manager-86878c769f-hh2pt\" (UID: \"ebfea797-c25e-4207-8101-94aa78d9d543\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt" Jul 15 12:21:50.634809 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:50.634709 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebfea797-c25e-4207-8101-94aa78d9d543-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hh2pt\" (UID: \"ebfea797-c25e-4207-8101-94aa78d9d543\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt" Jul 15 12:21:50.735259 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:50.735227 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebfea797-c25e-4207-8101-94aa78d9d543-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hh2pt\" (UID: \"ebfea797-c25e-4207-8101-94aa78d9d543\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt" Jul 15 12:21:50.735389 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:50.735289 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6rjq\" (UniqueName: \"kubernetes.io/projected/ebfea797-c25e-4207-8101-94aa78d9d543-kube-api-access-z6rjq\") pod \"kuadrant-operator-controller-manager-86878c769f-hh2pt\" (UID: \"ebfea797-c25e-4207-8101-94aa78d9d543\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt" Jul 15 12:21:50.735575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:50.735558 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebfea797-c25e-4207-8101-94aa78d9d543-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hh2pt\" (UID: \"ebfea797-c25e-4207-8101-94aa78d9d543\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt" Jul 15 12:21:50.746589 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:50.746561 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6rjq\" (UniqueName: \"kubernetes.io/projected/ebfea797-c25e-4207-8101-94aa78d9d543-kube-api-access-z6rjq\") pod \"kuadrant-operator-controller-manager-86878c769f-hh2pt\" (UID: \"ebfea797-c25e-4207-8101-94aa78d9d543\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt" Jul 15 12:21:50.909872 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:50.909756 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:50.925560 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:50.925531 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt" Jul 15 12:21:51.011843 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:51.011755 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:51.056792 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:51.056746 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt"] Jul 15 12:21:51.060178 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:21:51.060144 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebfea797_c25e_4207_8101_94aa78d9d543.slice/crio-7b7cfd1b1b7ad6cdbfd0bdec56ae6ca73031ccd9e828f29edff6cd7de5958a74 WatchSource:0}: Error finding container 7b7cfd1b1b7ad6cdbfd0bdec56ae6ca73031ccd9e828f29edff6cd7de5958a74: Status 404 returned error can't find the container with id 7b7cfd1b1b7ad6cdbfd0bdec56ae6ca73031ccd9e828f29edff6cd7de5958a74 Jul 15 12:21:51.070580 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:51.070557 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt" event={"ID":"ebfea797-c25e-4207-8101-94aa78d9d543","Type":"ContainerStarted","Data":"7b7cfd1b1b7ad6cdbfd0bdec56ae6ca73031ccd9e828f29edff6cd7de5958a74"} Jul 15 12:21:52.076567 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:52.076535 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt" event={"ID":"ebfea797-c25e-4207-8101-94aa78d9d543","Type":"ContainerStarted","Data":"565c78c24985488a840b15ae54fbe0977bf2ade000f3a699f01c25ab34082643"} Jul 15 12:21:52.076983 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:52.076657 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt" Jul 15 12:21:52.100884 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:52.100842 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt" podStartSLOduration=2.100827023 podStartE2EDuration="2.100827023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:21:52.100640328 +0000 UTC m=+635.735035276" watchObservedRunningTime="2026-07-15 12:21:52.100827023 +0000 UTC m=+635.735221970" Jul 15 12:21:54.504523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:54.504482 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:54.604434 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:21:54.604384 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:22:02.306609 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:22:02.306572 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:22:02.407425 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:22:02.407389 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:22:03.082886 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:22:03.082855 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt" Jul 15 12:22:06.961405 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:22:06.961366 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:22:07.452157 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:22:07.452121 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:22:07.699506 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:22:07.699475 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:22:07.748648 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:22:07.748619 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:22:08.315754 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:22:08.315718 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:22:08.369018 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:22:08.368992 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:22:23.451974 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:22:23.451936 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:22:23.568622 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:22:23.568587 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:22:23.623223 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:22:23.623188 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:22:24.274577 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:22:24.274543 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:22:24.341145 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:22:24.341112 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:22:40.354407 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:22:40.354379 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:22:40.453997 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:22:40.453961 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:22:40.910761 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:22:40.910726 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:22:53.572470 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:22:53.572434 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:22:53.736593 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:22:53.736560 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:00.469818 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:00.469760 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:00.566376 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:00.566340 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:00.669569 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:00.669533 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:00.691073 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:00.691044 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-69bf4cdfcb-tfvrx"] Jul 15 12:23:00.693177 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:00.693162 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69bf4cdfcb-tfvrx" Jul 15 12:23:00.695727 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:00.695704 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 15 12:23:00.695727 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:00.695715 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 15 12:23:00.695955 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:00.695935 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jfv4f\"" Jul 15 12:23:00.701681 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:00.701658 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69bf4cdfcb-tfvrx"] Jul 15 12:23:00.766584 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:00.766562 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbnf2\" (UniqueName: \"kubernetes.io/projected/e96585d0-2bba-4c6b-9e6f-701440653d66-kube-api-access-fbnf2\") pod \"maas-controller-69bf4cdfcb-tfvrx\" (UID: \"e96585d0-2bba-4c6b-9e6f-701440653d66\") " pod="opendatahub/maas-controller-69bf4cdfcb-tfvrx" Jul 15 12:23:00.766711 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:00.766599 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e96585d0-2bba-4c6b-9e6f-701440653d66-webhook-cert\") pod \"maas-controller-69bf4cdfcb-tfvrx\" (UID: \"e96585d0-2bba-4c6b-9e6f-701440653d66\") " pod="opendatahub/maas-controller-69bf4cdfcb-tfvrx" Jul 15 12:23:00.867630 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:00.867602 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbnf2\" (UniqueName: \"kubernetes.io/projected/e96585d0-2bba-4c6b-9e6f-701440653d66-kube-api-access-fbnf2\") pod \"maas-controller-69bf4cdfcb-tfvrx\" (UID: \"e96585d0-2bba-4c6b-9e6f-701440653d66\") " pod="opendatahub/maas-controller-69bf4cdfcb-tfvrx" Jul 15 12:23:00.867763 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:00.867663 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e96585d0-2bba-4c6b-9e6f-701440653d66-webhook-cert\") pod \"maas-controller-69bf4cdfcb-tfvrx\" (UID: \"e96585d0-2bba-4c6b-9e6f-701440653d66\") " pod="opendatahub/maas-controller-69bf4cdfcb-tfvrx" Jul 15 12:23:00.867844 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:23:00.867829 2556 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 15 12:23:00.867895 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:23:00.867885 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e96585d0-2bba-4c6b-9e6f-701440653d66-webhook-cert podName:e96585d0-2bba-4c6b-9e6f-701440653d66 nodeName:}" failed. No retries permitted until 2026-07-15 12:23:01.367869912 +0000 UTC m=+705.002264839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/e96585d0-2bba-4c6b-9e6f-701440653d66-webhook-cert") pod "maas-controller-69bf4cdfcb-tfvrx" (UID: "e96585d0-2bba-4c6b-9e6f-701440653d66") : secret "maas-controller-webhook-cert" not found Jul 15 12:23:00.876906 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:00.876883 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbnf2\" (UniqueName: \"kubernetes.io/projected/e96585d0-2bba-4c6b-9e6f-701440653d66-kube-api-access-fbnf2\") pod \"maas-controller-69bf4cdfcb-tfvrx\" (UID: \"e96585d0-2bba-4c6b-9e6f-701440653d66\") " pod="opendatahub/maas-controller-69bf4cdfcb-tfvrx" Jul 15 12:23:01.061373 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:01.061282 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:01.372210 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:01.372128 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e96585d0-2bba-4c6b-9e6f-701440653d66-webhook-cert\") pod \"maas-controller-69bf4cdfcb-tfvrx\" (UID: \"e96585d0-2bba-4c6b-9e6f-701440653d66\") " pod="opendatahub/maas-controller-69bf4cdfcb-tfvrx" Jul 15 12:23:01.374698 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:01.374674 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e96585d0-2bba-4c6b-9e6f-701440653d66-webhook-cert\") pod \"maas-controller-69bf4cdfcb-tfvrx\" (UID: \"e96585d0-2bba-4c6b-9e6f-701440653d66\") " pod="opendatahub/maas-controller-69bf4cdfcb-tfvrx" Jul 15 12:23:01.605028 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:01.604996 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69bf4cdfcb-tfvrx" Jul 15 12:23:01.730500 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:01.730478 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69bf4cdfcb-tfvrx"] Jul 15 12:23:01.733187 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:23:01.733158 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96585d0_2bba_4c6b_9e6f_701440653d66.slice/crio-4018efa4b8fe028021196286197ce23a03f0d500d6d3d8160196475f06b0e908 WatchSource:0}: Error finding container 4018efa4b8fe028021196286197ce23a03f0d500d6d3d8160196475f06b0e908: Status 404 returned error can't find the container with id 4018efa4b8fe028021196286197ce23a03f0d500d6d3d8160196475f06b0e908 Jul 15 12:23:02.359827 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:02.359757 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69bf4cdfcb-tfvrx" event={"ID":"e96585d0-2bba-4c6b-9e6f-701440653d66","Type":"ContainerStarted","Data":"4018efa4b8fe028021196286197ce23a03f0d500d6d3d8160196475f06b0e908"} Jul 15 12:23:04.268301 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:04.268253 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:04.369465 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:04.369382 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69bf4cdfcb-tfvrx" event={"ID":"e96585d0-2bba-4c6b-9e6f-701440653d66","Type":"ContainerStarted","Data":"7e5e873075a178751c750596a9cd1bcadb68db08c41512d657acffb2695b4ed7"} Jul 15 12:23:04.369623 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:04.369509 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-69bf4cdfcb-tfvrx" Jul 15 12:23:04.405051 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:04.405011 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:04.413492 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:04.413438 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-69bf4cdfcb-tfvrx" podStartSLOduration=2.079291884 podStartE2EDuration="4.413418557s" totalImagesPullingTime="2.334126673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:23:04.411164217 +0000 UTC m=+708.045559166" watchObservedRunningTime="2026-07-15 12:23:04.413418557 +0000 UTC m=+708.047813507" Jul 15 12:23:15.375527 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:15.375492 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-69bf4cdfcb-tfvrx" Jul 15 12:23:33.515233 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:33.515194 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:33.765804 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:33.765690 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:34.000542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:34.000511 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:34.158743 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:34.158645 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:34.475636 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:34.475558 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:34.622301 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:34.622268 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:40.035372 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:40.035284 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:45.108748 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:45.108707 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:45.208661 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:45.208583 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:55.609412 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:23:55.609375 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:24:03.947439 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:03.947408 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6"] Jul 15 12:24:03.950671 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:03.950650 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:03.953355 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:03.953333 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 12:24:03.953448 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:03.953405 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 12:24:03.954810 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:03.954793 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-p4ggq\"" Jul 15 12:24:03.954860 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:03.954830 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 15 12:24:03.961762 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:03.961740 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6"] Jul 15 12:24:04.042605 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.042565 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:24:04.062066 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.062041 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e7ea4ff-100c-420e-ad9e-ad1327152bd1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-h79x6\" (UID: \"9e7ea4ff-100c-420e-ad9e-ad1327152bd1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:04.062204 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.062080 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e7ea4ff-100c-420e-ad9e-ad1327152bd1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-h79x6\" (UID: \"9e7ea4ff-100c-420e-ad9e-ad1327152bd1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:04.062204 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.062107 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e7ea4ff-100c-420e-ad9e-ad1327152bd1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-h79x6\" (UID: \"9e7ea4ff-100c-420e-ad9e-ad1327152bd1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:04.062204 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.062135 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxt62\" (UniqueName: \"kubernetes.io/projected/9e7ea4ff-100c-420e-ad9e-ad1327152bd1-kube-api-access-gxt62\") pod \"e2e-distinct-simulated-kserve-59975d69f9-h79x6\" (UID: \"9e7ea4ff-100c-420e-ad9e-ad1327152bd1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:04.062204 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.062168 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e7ea4ff-100c-420e-ad9e-ad1327152bd1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-h79x6\" (UID: \"9e7ea4ff-100c-420e-ad9e-ad1327152bd1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:04.062204 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.062182 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e7ea4ff-100c-420e-ad9e-ad1327152bd1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-h79x6\" (UID: \"9e7ea4ff-100c-420e-ad9e-ad1327152bd1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:04.162702 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.162669 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e7ea4ff-100c-420e-ad9e-ad1327152bd1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-h79x6\" (UID: \"9e7ea4ff-100c-420e-ad9e-ad1327152bd1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:04.162909 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.162725 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e7ea4ff-100c-420e-ad9e-ad1327152bd1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-h79x6\" (UID: \"9e7ea4ff-100c-420e-ad9e-ad1327152bd1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:04.162909 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.162764 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e7ea4ff-100c-420e-ad9e-ad1327152bd1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-h79x6\" (UID: \"9e7ea4ff-100c-420e-ad9e-ad1327152bd1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:04.162909 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.162823 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxt62\" (UniqueName: \"kubernetes.io/projected/9e7ea4ff-100c-420e-ad9e-ad1327152bd1-kube-api-access-gxt62\") pod \"e2e-distinct-simulated-kserve-59975d69f9-h79x6\" (UID: \"9e7ea4ff-100c-420e-ad9e-ad1327152bd1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:04.162909 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.162868 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e7ea4ff-100c-420e-ad9e-ad1327152bd1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-h79x6\" (UID: \"9e7ea4ff-100c-420e-ad9e-ad1327152bd1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:04.162909 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.162892 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e7ea4ff-100c-420e-ad9e-ad1327152bd1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-h79x6\" (UID: \"9e7ea4ff-100c-420e-ad9e-ad1327152bd1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:04.163162 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.163126 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e7ea4ff-100c-420e-ad9e-ad1327152bd1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-h79x6\" (UID: \"9e7ea4ff-100c-420e-ad9e-ad1327152bd1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:04.163249 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.163220 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e7ea4ff-100c-420e-ad9e-ad1327152bd1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-h79x6\" (UID: \"9e7ea4ff-100c-420e-ad9e-ad1327152bd1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:04.163475 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.163440 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e7ea4ff-100c-420e-ad9e-ad1327152bd1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-h79x6\" (UID: \"9e7ea4ff-100c-420e-ad9e-ad1327152bd1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:04.165320 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.165295 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e7ea4ff-100c-420e-ad9e-ad1327152bd1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-h79x6\" (UID: \"9e7ea4ff-100c-420e-ad9e-ad1327152bd1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:04.165747 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.165728 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e7ea4ff-100c-420e-ad9e-ad1327152bd1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-h79x6\" (UID: \"9e7ea4ff-100c-420e-ad9e-ad1327152bd1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:04.170930 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.170911 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxt62\" (UniqueName: \"kubernetes.io/projected/9e7ea4ff-100c-420e-ad9e-ad1327152bd1-kube-api-access-gxt62\") pod \"e2e-distinct-simulated-kserve-59975d69f9-h79x6\" (UID: \"9e7ea4ff-100c-420e-ad9e-ad1327152bd1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:04.260807 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.260748 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:04.391218 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.391189 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6"] Jul 15 12:24:04.394037 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:24:04.394008 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7ea4ff_100c_420e_ad9e_ad1327152bd1.slice/crio-25c2db500870e8a9a68e24876507269a7535a6d1e671ace6540927f884c2ac32 WatchSource:0}: Error finding container 25c2db500870e8a9a68e24876507269a7535a6d1e671ace6540927f884c2ac32: Status 404 returned error can't find the container with id 25c2db500870e8a9a68e24876507269a7535a6d1e671ace6540927f884c2ac32 Jul 15 12:24:04.395822 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.395803 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 12:24:04.634261 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:04.634139 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" event={"ID":"9e7ea4ff-100c-420e-ad9e-ad1327152bd1","Type":"ContainerStarted","Data":"25c2db500870e8a9a68e24876507269a7535a6d1e671ace6540927f884c2ac32"} Jul 15 12:24:05.639411 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:05.639377 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" event={"ID":"9e7ea4ff-100c-420e-ad9e-ad1327152bd1","Type":"ContainerStarted","Data":"85166928679f040b13f832da6f919197682849086f91167d7524ddaaec5b6a15"} Jul 15 12:24:05.657630 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:05.657585 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" podStartSLOduration=1.593910277 podStartE2EDuration="2.65756667s" totalImagesPullingTime="1.063656393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:24:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:24:05.657454025 +0000 UTC m=+769.291848974" watchObservedRunningTime="2026-07-15 12:24:05.65756667 +0000 UTC m=+769.291961619" Jul 15 12:24:14.261382 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:14.261345 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:14.261816 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:14.261408 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:14.273732 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:14.273709 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:14.688387 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:14.688311 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-h79x6" Jul 15 12:24:15.085904 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:15.085863 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:24:42.700851 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:42.700740 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:24:47.045415 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.045381 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh"] Jul 15 12:24:47.047878 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.047850 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:47.050886 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.050859 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 15 12:24:47.058506 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.058485 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh"] Jul 15 12:24:47.070233 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.070204 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:24:47.076314 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.076275 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea3b2c7-baed-4588-8f65-e23603f2a68b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgvnh\" (UID: \"5ea3b2c7-baed-4588-8f65-e23603f2a68b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:47.076430 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.076360 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ea3b2c7-baed-4588-8f65-e23603f2a68b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgvnh\" (UID: \"5ea3b2c7-baed-4588-8f65-e23603f2a68b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:47.076430 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.076393 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7ff4\" (UniqueName: \"kubernetes.io/projected/5ea3b2c7-baed-4588-8f65-e23603f2a68b-kube-api-access-b7ff4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgvnh\" (UID: \"5ea3b2c7-baed-4588-8f65-e23603f2a68b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:47.076542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.076429 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ea3b2c7-baed-4588-8f65-e23603f2a68b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgvnh\" (UID: \"5ea3b2c7-baed-4588-8f65-e23603f2a68b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:47.076542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.076498 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ea3b2c7-baed-4588-8f65-e23603f2a68b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgvnh\" (UID: \"5ea3b2c7-baed-4588-8f65-e23603f2a68b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:47.076542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.076527 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea3b2c7-baed-4588-8f65-e23603f2a68b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgvnh\" (UID: \"5ea3b2c7-baed-4588-8f65-e23603f2a68b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:47.177272 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.177239 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ea3b2c7-baed-4588-8f65-e23603f2a68b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgvnh\" (UID: \"5ea3b2c7-baed-4588-8f65-e23603f2a68b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:47.177414 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.177294 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ea3b2c7-baed-4588-8f65-e23603f2a68b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgvnh\" (UID: \"5ea3b2c7-baed-4588-8f65-e23603f2a68b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:47.177414 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.177328 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea3b2c7-baed-4588-8f65-e23603f2a68b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgvnh\" (UID: \"5ea3b2c7-baed-4588-8f65-e23603f2a68b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:47.177414 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.177352 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea3b2c7-baed-4588-8f65-e23603f2a68b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgvnh\" (UID: \"5ea3b2c7-baed-4588-8f65-e23603f2a68b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:47.177414 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.177399 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ea3b2c7-baed-4588-8f65-e23603f2a68b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgvnh\" (UID: \"5ea3b2c7-baed-4588-8f65-e23603f2a68b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:47.177625 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.177428 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7ff4\" (UniqueName: \"kubernetes.io/projected/5ea3b2c7-baed-4588-8f65-e23603f2a68b-kube-api-access-b7ff4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgvnh\" (UID: \"5ea3b2c7-baed-4588-8f65-e23603f2a68b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:47.177711 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.177690 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ea3b2c7-baed-4588-8f65-e23603f2a68b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgvnh\" (UID: \"5ea3b2c7-baed-4588-8f65-e23603f2a68b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:47.177811 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.177793 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ea3b2c7-baed-4588-8f65-e23603f2a68b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgvnh\" (UID: \"5ea3b2c7-baed-4588-8f65-e23603f2a68b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:47.177870 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.177846 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea3b2c7-baed-4588-8f65-e23603f2a68b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgvnh\" (UID: \"5ea3b2c7-baed-4588-8f65-e23603f2a68b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:47.179816 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.179795 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ea3b2c7-baed-4588-8f65-e23603f2a68b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgvnh\" (UID: \"5ea3b2c7-baed-4588-8f65-e23603f2a68b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:47.180086 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.180067 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea3b2c7-baed-4588-8f65-e23603f2a68b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgvnh\" (UID: \"5ea3b2c7-baed-4588-8f65-e23603f2a68b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:47.185544 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.185518 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7ff4\" (UniqueName: \"kubernetes.io/projected/5ea3b2c7-baed-4588-8f65-e23603f2a68b-kube-api-access-b7ff4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgvnh\" (UID: \"5ea3b2c7-baed-4588-8f65-e23603f2a68b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:47.359238 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.359141 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:47.498059 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.498036 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh"] Jul 15 12:24:47.500495 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:24:47.500462 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea3b2c7_baed_4588_8f65_e23603f2a68b.slice/crio-8ead1159eaed966e0327245ffb603861ec8d14e02e2b7300dd2fe7f1b8175f34 WatchSource:0}: Error finding container 8ead1159eaed966e0327245ffb603861ec8d14e02e2b7300dd2fe7f1b8175f34: Status 404 returned error can't find the container with id 8ead1159eaed966e0327245ffb603861ec8d14e02e2b7300dd2fe7f1b8175f34 Jul 15 12:24:47.824111 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.824064 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" event={"ID":"5ea3b2c7-baed-4588-8f65-e23603f2a68b","Type":"ContainerStarted","Data":"1d6e7d7f0b17f4f049b21fb2d3114df16bde14a9d62eda14f1016a579921d09d"} Jul 15 12:24:47.824340 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.824130 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" event={"ID":"5ea3b2c7-baed-4588-8f65-e23603f2a68b","Type":"ContainerStarted","Data":"8ead1159eaed966e0327245ffb603861ec8d14e02e2b7300dd2fe7f1b8175f34"} Jul 15 12:24:47.842405 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:47.842358 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" podStartSLOduration=0.62180069 podStartE2EDuration="842.341197ms" totalImagesPullingTime="220.540507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:24:47.840336321 +0000 UTC m=+811.474731271" watchObservedRunningTime="2026-07-15 12:24:47.842341197 +0000 UTC m=+811.476736146" Jul 15 12:24:57.360321 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:57.360279 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:57.360756 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:57.360348 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:57.372369 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:57.372349 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:24:57.875630 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:24:57.875599 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgvnh" Jul 15 12:25:00.785622 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:25:00.785586 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:25:43.339557 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:25:43.339512 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:25:43.387835 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:25:43.387805 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:25:47.109404 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:25:47.109362 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:26:16.747066 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:26:16.747033 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:26:16.748666 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:26:16.748643 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:26:19.687667 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:26:19.687625 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:26:20.839939 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:26:20.839901 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:26:20.935728 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:26:20.935695 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:26:53.528266 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:26:53.528231 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:27:09.719535 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:09.719498 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:27:25.048079 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:25.048040 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:27:25.097095 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:25.097064 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:27:38.424394 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:38.424308 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:27:40.083222 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:40.083180 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:27:40.133163 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:40.133128 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:27:53.545803 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:53.545737 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:27:53.655076 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:53.655047 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69bf4cdfcb-tfvrx"] Jul 15 12:27:53.655274 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:53.655249 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-69bf4cdfcb-tfvrx" podUID="e96585d0-2bba-4c6b-9e6f-701440653d66" containerName="manager" containerID="cri-o://7e5e873075a178751c750596a9cd1bcadb68db08c41512d657acffb2695b4ed7" gracePeriod=30 Jul 15 12:27:53.891211 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:53.891184 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69bf4cdfcb-tfvrx" Jul 15 12:27:53.973103 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:53.973073 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e96585d0-2bba-4c6b-9e6f-701440653d66-webhook-cert\") pod \"e96585d0-2bba-4c6b-9e6f-701440653d66\" (UID: \"e96585d0-2bba-4c6b-9e6f-701440653d66\") " Jul 15 12:27:53.973259 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:53.973164 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbnf2\" (UniqueName: \"kubernetes.io/projected/e96585d0-2bba-4c6b-9e6f-701440653d66-kube-api-access-fbnf2\") pod \"e96585d0-2bba-4c6b-9e6f-701440653d66\" (UID: \"e96585d0-2bba-4c6b-9e6f-701440653d66\") " Jul 15 12:27:53.975219 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:53.975184 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e96585d0-2bba-4c6b-9e6f-701440653d66-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e96585d0-2bba-4c6b-9e6f-701440653d66" (UID: "e96585d0-2bba-4c6b-9e6f-701440653d66"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:27:53.975343 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:53.975304 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96585d0-2bba-4c6b-9e6f-701440653d66-kube-api-access-fbnf2" (OuterVolumeSpecName: "kube-api-access-fbnf2") pod "e96585d0-2bba-4c6b-9e6f-701440653d66" (UID: "e96585d0-2bba-4c6b-9e6f-701440653d66"). InnerVolumeSpecName "kube-api-access-fbnf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:27:54.074354 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:54.074282 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbnf2\" (UniqueName: \"kubernetes.io/projected/e96585d0-2bba-4c6b-9e6f-701440653d66-kube-api-access-fbnf2\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:27:54.074354 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:54.074309 2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e96585d0-2bba-4c6b-9e6f-701440653d66-webhook-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:27:54.588601 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:54.588569 2556 generic.go:356] "Generic (PLEG): container finished" podID="e96585d0-2bba-4c6b-9e6f-701440653d66" containerID="7e5e873075a178751c750596a9cd1bcadb68db08c41512d657acffb2695b4ed7" exitCode=0 Jul 15 12:27:54.589028 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:54.588611 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69bf4cdfcb-tfvrx" Jul 15 12:27:54.589028 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:54.588638 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69bf4cdfcb-tfvrx" event={"ID":"e96585d0-2bba-4c6b-9e6f-701440653d66","Type":"ContainerDied","Data":"7e5e873075a178751c750596a9cd1bcadb68db08c41512d657acffb2695b4ed7"} Jul 15 12:27:54.589028 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:54.588681 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69bf4cdfcb-tfvrx" event={"ID":"e96585d0-2bba-4c6b-9e6f-701440653d66","Type":"ContainerDied","Data":"4018efa4b8fe028021196286197ce23a03f0d500d6d3d8160196475f06b0e908"} Jul 15 12:27:54.589028 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:54.588707 2556 scope.go:117] "RemoveContainer" containerID="7e5e873075a178751c750596a9cd1bcadb68db08c41512d657acffb2695b4ed7" Jul 15 12:27:54.597757 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:54.597737 2556 scope.go:117] "RemoveContainer" containerID="7e5e873075a178751c750596a9cd1bcadb68db08c41512d657acffb2695b4ed7" Jul 15 12:27:54.598040 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:27:54.598019 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e5e873075a178751c750596a9cd1bcadb68db08c41512d657acffb2695b4ed7\": container with ID starting with 7e5e873075a178751c750596a9cd1bcadb68db08c41512d657acffb2695b4ed7 not found: ID does not exist" containerID="7e5e873075a178751c750596a9cd1bcadb68db08c41512d657acffb2695b4ed7" Jul 15 12:27:54.598091 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:54.598048 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e5e873075a178751c750596a9cd1bcadb68db08c41512d657acffb2695b4ed7"} err="failed to get container status \"7e5e873075a178751c750596a9cd1bcadb68db08c41512d657acffb2695b4ed7\": rpc error: code = NotFound desc = could not find container \"7e5e873075a178751c750596a9cd1bcadb68db08c41512d657acffb2695b4ed7\": container with ID starting with 7e5e873075a178751c750596a9cd1bcadb68db08c41512d657acffb2695b4ed7 not found: ID does not exist" Jul 15 12:27:54.611678 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:54.611648 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69bf4cdfcb-tfvrx"] Jul 15 12:27:54.615088 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:54.615067 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-69bf4cdfcb-tfvrx"] Jul 15 12:27:54.857002 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:54.854837 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e96585d0-2bba-4c6b-9e6f-701440653d66" path="/var/lib/kubelet/pods/e96585d0-2bba-4c6b-9e6f-701440653d66/volumes" Jul 15 12:27:55.286846 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:27:55.286807 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:28:41.128437 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:28:41.128390 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:28:41.252111 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:28:41.252073 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:28:41.393612 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:28:41.393524 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:28:47.189697 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:28:47.189664 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:28:47.259992 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:28:47.259955 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:28:47.336026 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:28:47.335999 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:28:47.435865 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:28:47.435833 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:28:47.677222 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:28:47.677193 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:28:53.326016 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:28:53.325983 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:28:53.390261 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:28:53.390221 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:31:16.769113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:31:16.769082 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:31:16.772139 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:31:16.772119 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:31:31.170015 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:31:31.169969 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:46.081219 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:32:46.081183 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt"] Jul 15 12:32:46.081691 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:32:46.081384 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt" podUID="ebfea797-c25e-4207-8101-94aa78d9d543" containerName="manager" containerID="cri-o://565c78c24985488a840b15ae54fbe0977bf2ade000f3a699f01c25ab34082643" gracePeriod=10 Jul 15 12:32:48.824615 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:32:48.824544 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:49.129874 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:32:49.129852 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt" Jul 15 12:32:49.148130 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:32:49.148111 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6rjq\" (UniqueName: \"kubernetes.io/projected/ebfea797-c25e-4207-8101-94aa78d9d543-kube-api-access-z6rjq\") pod \"ebfea797-c25e-4207-8101-94aa78d9d543\" (UID: \"ebfea797-c25e-4207-8101-94aa78d9d543\") " Jul 15 12:32:49.148243 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:32:49.148175 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebfea797-c25e-4207-8101-94aa78d9d543-extensions-socket-volume\") pod \"ebfea797-c25e-4207-8101-94aa78d9d543\" (UID: \"ebfea797-c25e-4207-8101-94aa78d9d543\") " Jul 15 12:32:49.148544 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:32:49.148522 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebfea797-c25e-4207-8101-94aa78d9d543-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ebfea797-c25e-4207-8101-94aa78d9d543" (UID: "ebfea797-c25e-4207-8101-94aa78d9d543"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:32:49.150644 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:32:49.150611 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebfea797-c25e-4207-8101-94aa78d9d543-kube-api-access-z6rjq" (OuterVolumeSpecName: "kube-api-access-z6rjq") pod "ebfea797-c25e-4207-8101-94aa78d9d543" (UID: "ebfea797-c25e-4207-8101-94aa78d9d543"). InnerVolumeSpecName "kube-api-access-z6rjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:32:49.249011 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:32:49.248964 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6rjq\" (UniqueName: \"kubernetes.io/projected/ebfea797-c25e-4207-8101-94aa78d9d543-kube-api-access-z6rjq\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:32:49.249011 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:32:49.249005 2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebfea797-c25e-4207-8101-94aa78d9d543-extensions-socket-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:32:49.821203 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:32:49.821170 2556 generic.go:356] "Generic (PLEG): container finished" podID="ebfea797-c25e-4207-8101-94aa78d9d543" containerID="565c78c24985488a840b15ae54fbe0977bf2ade000f3a699f01c25ab34082643" exitCode=0 Jul 15 12:32:49.821369 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:32:49.821213 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt" Jul 15 12:32:49.821369 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:32:49.821251 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt" event={"ID":"ebfea797-c25e-4207-8101-94aa78d9d543","Type":"ContainerDied","Data":"565c78c24985488a840b15ae54fbe0977bf2ade000f3a699f01c25ab34082643"} Jul 15 12:32:49.821369 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:32:49.821300 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt" event={"ID":"ebfea797-c25e-4207-8101-94aa78d9d543","Type":"ContainerDied","Data":"7b7cfd1b1b7ad6cdbfd0bdec56ae6ca73031ccd9e828f29edff6cd7de5958a74"} Jul 15 12:32:49.821369 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:32:49.821323 2556 scope.go:117] "RemoveContainer" containerID="565c78c24985488a840b15ae54fbe0977bf2ade000f3a699f01c25ab34082643" Jul 15 12:32:49.830850 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:32:49.830827 2556 scope.go:117] "RemoveContainer" containerID="565c78c24985488a840b15ae54fbe0977bf2ade000f3a699f01c25ab34082643" Jul 15 12:32:49.831124 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:32:49.831089 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"565c78c24985488a840b15ae54fbe0977bf2ade000f3a699f01c25ab34082643\": container with ID starting with 565c78c24985488a840b15ae54fbe0977bf2ade000f3a699f01c25ab34082643 not found: ID does not exist" containerID="565c78c24985488a840b15ae54fbe0977bf2ade000f3a699f01c25ab34082643" Jul 15 12:32:49.831124 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:32:49.831112 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"565c78c24985488a840b15ae54fbe0977bf2ade000f3a699f01c25ab34082643"} err="failed to get container status \"565c78c24985488a840b15ae54fbe0977bf2ade000f3a699f01c25ab34082643\": rpc error: code = NotFound desc = could not find container \"565c78c24985488a840b15ae54fbe0977bf2ade000f3a699f01c25ab34082643\": container with ID starting with 565c78c24985488a840b15ae54fbe0977bf2ade000f3a699f01c25ab34082643 not found: ID does not exist" Jul 15 12:32:49.841185 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:32:49.841165 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt"] Jul 15 12:32:49.844914 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:32:49.844894 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hh2pt"] Jul 15 12:32:50.853582 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:32:50.853548 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebfea797-c25e-4207-8101-94aa78d9d543" path="/var/lib/kubelet/pods/ebfea797-c25e-4207-8101-94aa78d9d543/volumes" Jul 15 12:33:47.258096 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:47.258013 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ktxhx"] Jul 15 12:33:47.258564 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:47.258294 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e96585d0-2bba-4c6b-9e6f-701440653d66" containerName="manager" Jul 15 12:33:47.258564 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:47.258303 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e96585d0-2bba-4c6b-9e6f-701440653d66" containerName="manager" Jul 15 12:33:47.258564 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:47.258316 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebfea797-c25e-4207-8101-94aa78d9d543" containerName="manager" Jul 15 12:33:47.258564 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:47.258322 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebfea797-c25e-4207-8101-94aa78d9d543" containerName="manager" Jul 15 12:33:47.258564 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:47.258375 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e96585d0-2bba-4c6b-9e6f-701440653d66" containerName="manager" Jul 15 12:33:47.258564 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:47.258384 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebfea797-c25e-4207-8101-94aa78d9d543" containerName="manager" Jul 15 12:33:47.260458 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:47.260443 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ktxhx" Jul 15 12:33:47.263307 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:47.263286 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jps4v\"" Jul 15 12:33:47.274849 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:47.274828 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ktxhx"] Jul 15 12:33:47.415939 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:47.415907 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3fe1727-a76d-4436-9b2c-6dab569e8e26-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ktxhx\" (UID: \"a3fe1727-a76d-4436-9b2c-6dab569e8e26\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ktxhx" Jul 15 12:33:47.416126 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:47.415970 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fhbt\" (UniqueName: \"kubernetes.io/projected/a3fe1727-a76d-4436-9b2c-6dab569e8e26-kube-api-access-4fhbt\") pod \"kuadrant-operator-controller-manager-86878c769f-ktxhx\" (UID: \"a3fe1727-a76d-4436-9b2c-6dab569e8e26\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ktxhx" Jul 15 12:33:47.517410 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:47.517318 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3fe1727-a76d-4436-9b2c-6dab569e8e26-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ktxhx\" (UID: \"a3fe1727-a76d-4436-9b2c-6dab569e8e26\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ktxhx" Jul 15 12:33:47.517410 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:47.517381 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fhbt\" (UniqueName: \"kubernetes.io/projected/a3fe1727-a76d-4436-9b2c-6dab569e8e26-kube-api-access-4fhbt\") pod \"kuadrant-operator-controller-manager-86878c769f-ktxhx\" (UID: \"a3fe1727-a76d-4436-9b2c-6dab569e8e26\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ktxhx" Jul 15 12:33:47.517709 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:47.517688 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3fe1727-a76d-4436-9b2c-6dab569e8e26-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ktxhx\" (UID: \"a3fe1727-a76d-4436-9b2c-6dab569e8e26\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ktxhx" Jul 15 12:33:47.529031 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:47.528998 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fhbt\" (UniqueName: \"kubernetes.io/projected/a3fe1727-a76d-4436-9b2c-6dab569e8e26-kube-api-access-4fhbt\") pod \"kuadrant-operator-controller-manager-86878c769f-ktxhx\" (UID: \"a3fe1727-a76d-4436-9b2c-6dab569e8e26\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ktxhx" Jul 15 12:33:47.570645 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:47.570607 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ktxhx" Jul 15 12:33:47.701830 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:47.701808 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ktxhx"] Jul 15 12:33:47.706887 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:33:47.706851 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3fe1727_a76d_4436_9b2c_6dab569e8e26.slice/crio-853ec19556741553062363afed1b768aacf047f63939cc17891a639b2f75de2d WatchSource:0}: Error finding container 853ec19556741553062363afed1b768aacf047f63939cc17891a639b2f75de2d: Status 404 returned error can't find the container with id 853ec19556741553062363afed1b768aacf047f63939cc17891a639b2f75de2d Jul 15 12:33:48.051230 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:48.051128 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ktxhx" event={"ID":"a3fe1727-a76d-4436-9b2c-6dab569e8e26","Type":"ContainerStarted","Data":"b93f186549c0b31e9290674c4d86762b024b6ca0b6cd82fc6bb13056daf1f5ba"} Jul 15 12:33:48.051230 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:48.051176 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ktxhx" event={"ID":"a3fe1727-a76d-4436-9b2c-6dab569e8e26","Type":"ContainerStarted","Data":"853ec19556741553062363afed1b768aacf047f63939cc17891a639b2f75de2d"} Jul 15 12:33:48.051446 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:48.051266 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ktxhx" Jul 15 12:33:48.075183 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:48.075121 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ktxhx" podStartSLOduration=1.075104185 podStartE2EDuration="1.075104185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:33:48.072826973 +0000 UTC m=+1351.707221915" watchObservedRunningTime="2026-07-15 12:33:48.075104185 +0000 UTC m=+1351.709499132" Jul 15 12:33:59.058895 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:33:59.058863 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ktxhx" Jul 15 12:34:27.285724 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:34:27.285691 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:27.375738 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:34:27.375700 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:16.792977 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:36:16.792949 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:36:16.797048 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:36:16.797027 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:36:41.959702 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:36:41.959622 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:47.775046 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:36:47.775008 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:30.250160 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:30.250124 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:39.183618 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:39.183542 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:39.284014 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:39.283985 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:39.644166 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:39.644131 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:45.387088 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:45.387056 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:45.484668 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:45.484639 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:45.618421 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:45.618371 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:45.712035 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:45.711938 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:46.788643 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:46.788607 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:46.889829 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:46.889801 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:46.948533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:46.948506 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:47.122355 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:47.122278 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:47.447807 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:47.447693 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:47.545048 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:47.545011 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:49.437420 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:49.437383 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:51.774568 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:51.774530 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e10a8061-z44x5"] Jul 15 12:39:51.777735 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:51.777713 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e10a8061-z44x5" Jul 15 12:39:51.780330 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:51.780308 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kv8zr\"" Jul 15 12:39:51.780425 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:51.780389 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 12:39:51.784473 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:51.784446 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e10a8061-z44x5"] Jul 15 12:39:51.858444 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:51.858416 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a890c67-9e3d-4c11-b61f-e7ef40b8c68d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e10a8061-z44x5\" (UID: \"9a890c67-9e3d-4c11-b61f-e7ef40b8c68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e10a8061-z44x5" Jul 15 12:39:51.959388 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:51.959358 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a890c67-9e3d-4c11-b61f-e7ef40b8c68d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e10a8061-z44x5\" (UID: \"9a890c67-9e3d-4c11-b61f-e7ef40b8c68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e10a8061-z44x5" Jul 15 12:39:51.959956 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:51.959937 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a890c67-9e3d-4c11-b61f-e7ef40b8c68d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e10a8061-z44x5\" (UID: \"9a890c67-9e3d-4c11-b61f-e7ef40b8c68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e10a8061-z44x5" Jul 15 12:39:52.089016 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:52.088950 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e10a8061-z44x5" Jul 15 12:39:52.226942 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:52.226916 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e10a8061-z44x5"] Jul 15 12:39:52.228673 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:39:52.228640 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a890c67_9e3d_4c11_b61f_e7ef40b8c68d.slice/crio-ce6455b20351fd936145aa9d471e6bbd18c9fd7eb058aff886e08fd1cedb656c WatchSource:0}: Error finding container ce6455b20351fd936145aa9d471e6bbd18c9fd7eb058aff886e08fd1cedb656c: Status 404 returned error can't find the container with id ce6455b20351fd936145aa9d471e6bbd18c9fd7eb058aff886e08fd1cedb656c Jul 15 12:39:52.230548 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:52.230532 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 12:39:52.555607 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:52.555573 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e10a8061-z44x5" event={"ID":"9a890c67-9e3d-4c11-b61f-e7ef40b8c68d","Type":"ContainerStarted","Data":"ce6455b20351fd936145aa9d471e6bbd18c9fd7eb058aff886e08fd1cedb656c"} Jul 15 12:39:55.571472 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:55.571444 2556 generic.go:356] "Generic (PLEG): container finished" podID="9a890c67-9e3d-4c11-b61f-e7ef40b8c68d" containerID="9e1c563261f0ba9038ab12c5606819915be6b485dfa528cd4c50ffd2c8ec18e6" exitCode=7 Jul 15 12:39:55.571848 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:55.571535 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e10a8061-z44x5" event={"ID":"9a890c67-9e3d-4c11-b61f-e7ef40b8c68d","Type":"ContainerDied","Data":"9e1c563261f0ba9038ab12c5606819915be6b485dfa528cd4c50ffd2c8ec18e6"} Jul 15 12:39:55.571848 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:55.571693 2556 scope.go:117] "RemoveContainer" containerID="9e1c563261f0ba9038ab12c5606819915be6b485dfa528cd4c50ffd2c8ec18e6" Jul 15 12:39:56.577845 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:56.577750 2556 generic.go:356] "Generic (PLEG): container finished" podID="9a890c67-9e3d-4c11-b61f-e7ef40b8c68d" containerID="f824a6a02e6cec46e61c6261642a06ffb6e8cb6fdaf7d3b731cb86397f2a9377" exitCode=0 Jul 15 12:39:56.578246 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:56.577842 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e10a8061-z44x5" event={"ID":"9a890c67-9e3d-4c11-b61f-e7ef40b8c68d","Type":"ContainerDied","Data":"f824a6a02e6cec46e61c6261642a06ffb6e8cb6fdaf7d3b731cb86397f2a9377"} Jul 15 12:39:56.578246 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:56.577904 2556 scope.go:117] "RemoveContainer" containerID="9e1c563261f0ba9038ab12c5606819915be6b485dfa528cd4c50ffd2c8ec18e6" Jul 15 12:39:57.716372 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:57.716347 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e10a8061-z44x5" Jul 15 12:39:57.810044 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:57.809996 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a890c67-9e3d-4c11-b61f-e7ef40b8c68d-maas-api-service-ca\") pod \"9a890c67-9e3d-4c11-b61f-e7ef40b8c68d\" (UID: \"9a890c67-9e3d-4c11-b61f-e7ef40b8c68d\") " Jul 15 12:39:57.810418 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:57.810389 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a890c67-9e3d-4c11-b61f-e7ef40b8c68d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a890c67-9e3d-4c11-b61f-e7ef40b8c68d" (UID: "9a890c67-9e3d-4c11-b61f-e7ef40b8c68d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:39:57.810512 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:57.810500 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a890c67-9e3d-4c11-b61f-e7ef40b8c68d-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:39:58.589441 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:58.589407 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e10a8061-z44x5" event={"ID":"9a890c67-9e3d-4c11-b61f-e7ef40b8c68d","Type":"ContainerDied","Data":"ce6455b20351fd936145aa9d471e6bbd18c9fd7eb058aff886e08fd1cedb656c"} Jul 15 12:39:58.589601 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:58.589454 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce6455b20351fd936145aa9d471e6bbd18c9fd7eb058aff886e08fd1cedb656c" Jul 15 12:39:58.589601 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:39:58.589413 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e10a8061-z44x5" Jul 15 12:40:02.186676 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:02.186637 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:06.859791 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:06.859736 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:12.468755 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:12.468713 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:12.569046 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:12.569014 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:17.073979 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:17.073951 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:17.457422 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:17.457343 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:17.603351 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:17.603318 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:17.806145 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:17.806115 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:18.994324 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:18.994296 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:19.184108 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.184079 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n"] Jul 15 12:40:19.184448 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.184430 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a890c67-9e3d-4c11-b61f-e7ef40b8c68d" containerName="revoke-keys" Jul 15 12:40:19.184528 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.184451 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a890c67-9e3d-4c11-b61f-e7ef40b8c68d" containerName="revoke-keys" Jul 15 12:40:19.184528 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.184466 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a890c67-9e3d-4c11-b61f-e7ef40b8c68d" containerName="revoke-keys" Jul 15 12:40:19.184528 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.184474 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a890c67-9e3d-4c11-b61f-e7ef40b8c68d" containerName="revoke-keys" Jul 15 12:40:19.184673 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.184560 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a890c67-9e3d-4c11-b61f-e7ef40b8c68d" containerName="revoke-keys" Jul 15 12:40:19.187625 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.187603 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" Jul 15 12:40:19.190422 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.190401 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-af71886a-serving-cert\"" Jul 15 12:40:19.190557 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.190446 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bnnf9\"" Jul 15 12:40:19.190557 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.190445 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 12:40:19.191533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.191503 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 12:40:19.197379 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.197358 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n"] Jul 15 12:40:19.217356 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.217324 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:19.298694 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.298654 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/228efbb0-01e0-4d1a-b55f-035dbecc5dad-openshift-service-ca\") pod \"maas-api-e2e-migrate-af71886a-64fcb58685-55q9n\" (UID: \"228efbb0-01e0-4d1a-b55f-035dbecc5dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" Jul 15 12:40:19.298694 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.298700 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdr67\" (UniqueName: \"kubernetes.io/projected/228efbb0-01e0-4d1a-b55f-035dbecc5dad-kube-api-access-pdr67\") pod \"maas-api-e2e-migrate-af71886a-64fcb58685-55q9n\" (UID: \"228efbb0-01e0-4d1a-b55f-035dbecc5dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" Jul 15 12:40:19.298961 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.298727 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/228efbb0-01e0-4d1a-b55f-035dbecc5dad-maas-api-tls\") pod \"maas-api-e2e-migrate-af71886a-64fcb58685-55q9n\" (UID: \"228efbb0-01e0-4d1a-b55f-035dbecc5dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" Jul 15 12:40:19.400169 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.400134 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/228efbb0-01e0-4d1a-b55f-035dbecc5dad-openshift-service-ca\") pod \"maas-api-e2e-migrate-af71886a-64fcb58685-55q9n\" (UID: \"228efbb0-01e0-4d1a-b55f-035dbecc5dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" Jul 15 12:40:19.400322 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.400197 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdr67\" (UniqueName: \"kubernetes.io/projected/228efbb0-01e0-4d1a-b55f-035dbecc5dad-kube-api-access-pdr67\") pod \"maas-api-e2e-migrate-af71886a-64fcb58685-55q9n\" (UID: \"228efbb0-01e0-4d1a-b55f-035dbecc5dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" Jul 15 12:40:19.400322 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.400238 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/228efbb0-01e0-4d1a-b55f-035dbecc5dad-maas-api-tls\") pod \"maas-api-e2e-migrate-af71886a-64fcb58685-55q9n\" (UID: \"228efbb0-01e0-4d1a-b55f-035dbecc5dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" Jul 15 12:40:19.400419 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:40:19.400405 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-serving-cert: secret "maas-api-e2e-migrate-af71886a-serving-cert" not found Jul 15 12:40:19.400488 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:40:19.400478 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/228efbb0-01e0-4d1a-b55f-035dbecc5dad-maas-api-tls podName:228efbb0-01e0-4d1a-b55f-035dbecc5dad nodeName:}" failed. No retries permitted until 2026-07-15 12:40:19.900456651 +0000 UTC m=+1743.534851594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/228efbb0-01e0-4d1a-b55f-035dbecc5dad-maas-api-tls") pod "maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" (UID: "228efbb0-01e0-4d1a-b55f-035dbecc5dad") : secret "maas-api-e2e-migrate-af71886a-serving-cert" not found Jul 15 12:40:19.400931 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.400903 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/228efbb0-01e0-4d1a-b55f-035dbecc5dad-openshift-service-ca\") pod \"maas-api-e2e-migrate-af71886a-64fcb58685-55q9n\" (UID: \"228efbb0-01e0-4d1a-b55f-035dbecc5dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" Jul 15 12:40:19.409320 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.409290 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdr67\" (UniqueName: \"kubernetes.io/projected/228efbb0-01e0-4d1a-b55f-035dbecc5dad-kube-api-access-pdr67\") pod \"maas-api-e2e-migrate-af71886a-64fcb58685-55q9n\" (UID: \"228efbb0-01e0-4d1a-b55f-035dbecc5dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" Jul 15 12:40:19.474384 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.474354 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:19.637278 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.637190 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:19.732178 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.732146 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:19.904861 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.904755 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/228efbb0-01e0-4d1a-b55f-035dbecc5dad-maas-api-tls\") pod \"maas-api-e2e-migrate-af71886a-64fcb58685-55q9n\" (UID: \"228efbb0-01e0-4d1a-b55f-035dbecc5dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" Jul 15 12:40:19.907388 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:19.907363 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/228efbb0-01e0-4d1a-b55f-035dbecc5dad-maas-api-tls\") pod \"maas-api-e2e-migrate-af71886a-64fcb58685-55q9n\" (UID: \"228efbb0-01e0-4d1a-b55f-035dbecc5dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" Jul 15 12:40:20.099180 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:20.099141 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" Jul 15 12:40:20.229099 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:20.229076 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n"] Jul 15 12:40:20.231151 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:40:20.231123 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228efbb0_01e0_4d1a_b55f_035dbecc5dad.slice/crio-d135c7475968b8dc185d8adc141a7a30e461604b607295057952c56b583eb9c8 WatchSource:0}: Error finding container d135c7475968b8dc185d8adc141a7a30e461604b607295057952c56b583eb9c8: Status 404 returned error can't find the container with id d135c7475968b8dc185d8adc141a7a30e461604b607295057952c56b583eb9c8 Jul 15 12:40:20.682827 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:20.682794 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" event={"ID":"228efbb0-01e0-4d1a-b55f-035dbecc5dad","Type":"ContainerStarted","Data":"d135c7475968b8dc185d8adc141a7a30e461604b607295057952c56b583eb9c8"} Jul 15 12:40:22.693656 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:22.693557 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" event={"ID":"228efbb0-01e0-4d1a-b55f-035dbecc5dad","Type":"ContainerStarted","Data":"cbd89b8072e444d104016e0e076bbd4d23e8efdd5588f4443e85a1279352d981"} Jul 15 12:40:22.694138 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:22.693670 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" Jul 15 12:40:22.710592 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:22.710544 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" podStartSLOduration=2.203876701 podStartE2EDuration="3.71052493s" totalImagesPullingTime="1.506648229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:40:22.709405309 +0000 UTC m=+1746.343800257" watchObservedRunningTime="2026-07-15 12:40:22.71052493 +0000 UTC m=+1746.344919878" Jul 15 12:40:23.430606 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:23.430571 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp"] Jul 15 12:40:23.430949 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:23.430933 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a890c67-9e3d-4c11-b61f-e7ef40b8c68d" containerName="revoke-keys" Jul 15 12:40:23.433934 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:23.433918 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp" Jul 15 12:40:23.436464 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:23.436443 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kv8zr\"" Jul 15 12:40:23.441498 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:23.441434 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp"] Jul 15 12:40:23.534361 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:23.534328 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/942767bc-3c91-44a3-9033-dd0ebf58a324-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp\" (UID: \"942767bc-3c91-44a3-9033-dd0ebf58a324\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp" Jul 15 12:40:23.635214 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:23.635177 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/942767bc-3c91-44a3-9033-dd0ebf58a324-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp\" (UID: \"942767bc-3c91-44a3-9033-dd0ebf58a324\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp" Jul 15 12:40:23.635806 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:23.635759 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/942767bc-3c91-44a3-9033-dd0ebf58a324-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp\" (UID: \"942767bc-3c91-44a3-9033-dd0ebf58a324\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp" Jul 15 12:40:23.744927 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:23.744848 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp" Jul 15 12:40:23.873593 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:23.873568 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp"] Jul 15 12:40:23.875538 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:40:23.875509 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942767bc_3c91_44a3_9033_dd0ebf58a324.slice/crio-0b29632382ad9c8696f2714aac2277977fd5c41fd4e06c987ca1788a1f1b2ab0 WatchSource:0}: Error finding container 0b29632382ad9c8696f2714aac2277977fd5c41fd4e06c987ca1788a1f1b2ab0: Status 404 returned error can't find the container with id 0b29632382ad9c8696f2714aac2277977fd5c41fd4e06c987ca1788a1f1b2ab0 Jul 15 12:40:24.066420 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:24.063717 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:24.111954 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:24.111923 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:24.706565 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:24.706533 2556 generic.go:356] "Generic (PLEG): container finished" podID="942767bc-3c91-44a3-9033-dd0ebf58a324" containerID="d93058b9af457855bb9295d64ce5097791628329909f674396ea93b63e52bbcc" exitCode=7 Jul 15 12:40:24.706735 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:24.706623 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp" event={"ID":"942767bc-3c91-44a3-9033-dd0ebf58a324","Type":"ContainerDied","Data":"d93058b9af457855bb9295d64ce5097791628329909f674396ea93b63e52bbcc"} Jul 15 12:40:24.706735 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:24.706702 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp" event={"ID":"942767bc-3c91-44a3-9033-dd0ebf58a324","Type":"ContainerStarted","Data":"0b29632382ad9c8696f2714aac2277977fd5c41fd4e06c987ca1788a1f1b2ab0"} Jul 15 12:40:24.706841 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:24.706814 2556 scope.go:117] "RemoveContainer" containerID="d93058b9af457855bb9295d64ce5097791628329909f674396ea93b63e52bbcc" Jul 15 12:40:25.713107 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:25.713072 2556 generic.go:356] "Generic (PLEG): container finished" podID="942767bc-3c91-44a3-9033-dd0ebf58a324" containerID="20185c6baf1c8546f6ef9ffc119faaf0cfa0739de7059a98c617c8e7dda9b3a4" exitCode=7 Jul 15 12:40:25.713502 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:25.713158 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp" event={"ID":"942767bc-3c91-44a3-9033-dd0ebf58a324","Type":"ContainerDied","Data":"20185c6baf1c8546f6ef9ffc119faaf0cfa0739de7059a98c617c8e7dda9b3a4"} Jul 15 12:40:25.713502 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:25.713218 2556 scope.go:117] "RemoveContainer" containerID="d93058b9af457855bb9295d64ce5097791628329909f674396ea93b63e52bbcc" Jul 15 12:40:25.713502 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:25.713473 2556 scope.go:117] "RemoveContainer" containerID="20185c6baf1c8546f6ef9ffc119faaf0cfa0739de7059a98c617c8e7dda9b3a4" Jul 15 12:40:25.713684 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:40:25.713666 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp_odh-ai-gateway-infra(942767bc-3c91-44a3-9033-dd0ebf58a324)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp" podUID="942767bc-3c91-44a3-9033-dd0ebf58a324" Jul 15 12:40:26.720682 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:26.720655 2556 scope.go:117] "RemoveContainer" containerID="20185c6baf1c8546f6ef9ffc119faaf0cfa0739de7059a98c617c8e7dda9b3a4" Jul 15 12:40:26.721102 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:40:26.720892 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp_odh-ai-gateway-infra(942767bc-3c91-44a3-9033-dd0ebf58a324)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp" podUID="942767bc-3c91-44a3-9033-dd0ebf58a324" Jul 15 12:40:27.678448 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:27.678419 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:28.703653 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:28.703625 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" Jul 15 12:40:35.480088 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:35.480056 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:35.532134 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:35.532105 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:41.848736 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:41.848698 2556 scope.go:117] "RemoveContainer" containerID="20185c6baf1c8546f6ef9ffc119faaf0cfa0739de7059a98c617c8e7dda9b3a4" Jul 15 12:40:42.788942 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:42.788912 2556 generic.go:356] "Generic (PLEG): container finished" podID="942767bc-3c91-44a3-9033-dd0ebf58a324" containerID="63d97e3bd599c54e4171768f17aa8325e4b6e56fd8043e857c30c1fff4c0b490" exitCode=0 Jul 15 12:40:42.789129 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:42.788991 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp" event={"ID":"942767bc-3c91-44a3-9033-dd0ebf58a324","Type":"ContainerDied","Data":"63d97e3bd599c54e4171768f17aa8325e4b6e56fd8043e857c30c1fff4c0b490"} Jul 15 12:40:42.789129 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:42.789065 2556 scope.go:117] "RemoveContainer" containerID="20185c6baf1c8546f6ef9ffc119faaf0cfa0739de7059a98c617c8e7dda9b3a4" Jul 15 12:40:43.818201 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:43.818171 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp"] Jul 15 12:40:43.924735 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:43.924710 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp" Jul 15 12:40:43.988000 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:43.987967 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/942767bc-3c91-44a3-9033-dd0ebf58a324-maas-api-service-ca\") pod \"942767bc-3c91-44a3-9033-dd0ebf58a324\" (UID: \"942767bc-3c91-44a3-9033-dd0ebf58a324\") " Jul 15 12:40:43.988276 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:43.988252 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942767bc-3c91-44a3-9033-dd0ebf58a324-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "942767bc-3c91-44a3-9033-dd0ebf58a324" (UID: "942767bc-3c91-44a3-9033-dd0ebf58a324"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:40:44.089382 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:44.089300 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/942767bc-3c91-44a3-9033-dd0ebf58a324-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:40:44.800801 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:44.800747 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp" Jul 15 12:40:44.800984 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:44.800807 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp" event={"ID":"942767bc-3c91-44a3-9033-dd0ebf58a324","Type":"ContainerDied","Data":"0b29632382ad9c8696f2714aac2277977fd5c41fd4e06c987ca1788a1f1b2ab0"} Jul 15 12:40:44.800984 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:44.800864 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b29632382ad9c8696f2714aac2277977fd5c41fd4e06c987ca1788a1f1b2ab0" Jul 15 12:40:44.827615 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:44.827588 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp"] Jul 15 12:40:44.833221 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:44.833200 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af71886a-d4wvp"] Jul 15 12:40:44.854007 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:40:44.853983 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="942767bc-3c91-44a3-9033-dd0ebf58a324" path="/var/lib/kubelet/pods/942767bc-3c91-44a3-9033-dd0ebf58a324/volumes" Jul 15 12:41:16.818749 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:16.818721 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:41:16.822525 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:16.822506 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:41:23.578865 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:23.578827 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:23.683075 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:23.683045 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:29.644893 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:29.644862 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n"] Jul 15 12:41:29.645426 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:29.645107 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" podUID="228efbb0-01e0-4d1a-b55f-035dbecc5dad" containerName="maas-api" containerID="cri-o://cbd89b8072e444d104016e0e076bbd4d23e8efdd5588f4443e85a1279352d981" gracePeriod=30 Jul 15 12:41:29.668417 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:29.668394 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:29.826518 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:29.826491 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:29.910380 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:29.910323 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" Jul 15 12:41:29.996213 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:29.996179 2556 generic.go:356] "Generic (PLEG): container finished" podID="228efbb0-01e0-4d1a-b55f-035dbecc5dad" containerID="cbd89b8072e444d104016e0e076bbd4d23e8efdd5588f4443e85a1279352d981" exitCode=0 Jul 15 12:41:29.996383 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:29.996238 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" Jul 15 12:41:29.996383 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:29.996241 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" event={"ID":"228efbb0-01e0-4d1a-b55f-035dbecc5dad","Type":"ContainerDied","Data":"cbd89b8072e444d104016e0e076bbd4d23e8efdd5588f4443e85a1279352d981"} Jul 15 12:41:29.996383 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:29.996350 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n" event={"ID":"228efbb0-01e0-4d1a-b55f-035dbecc5dad","Type":"ContainerDied","Data":"d135c7475968b8dc185d8adc141a7a30e461604b607295057952c56b583eb9c8"} Jul 15 12:41:29.996383 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:29.996377 2556 scope.go:117] "RemoveContainer" containerID="cbd89b8072e444d104016e0e076bbd4d23e8efdd5588f4443e85a1279352d981" Jul 15 12:41:30.005274 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:30.005249 2556 scope.go:117] "RemoveContainer" containerID="cbd89b8072e444d104016e0e076bbd4d23e8efdd5588f4443e85a1279352d981" Jul 15 12:41:30.005554 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:41:30.005528 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd89b8072e444d104016e0e076bbd4d23e8efdd5588f4443e85a1279352d981\": container with ID starting with cbd89b8072e444d104016e0e076bbd4d23e8efdd5588f4443e85a1279352d981 not found: ID does not exist" containerID="cbd89b8072e444d104016e0e076bbd4d23e8efdd5588f4443e85a1279352d981" Jul 15 12:41:30.005630 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:30.005562 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd89b8072e444d104016e0e076bbd4d23e8efdd5588f4443e85a1279352d981"} err="failed to get container status \"cbd89b8072e444d104016e0e076bbd4d23e8efdd5588f4443e85a1279352d981\": rpc error: code = NotFound desc = could not find container \"cbd89b8072e444d104016e0e076bbd4d23e8efdd5588f4443e85a1279352d981\": container with ID starting with cbd89b8072e444d104016e0e076bbd4d23e8efdd5588f4443e85a1279352d981 not found: ID does not exist" Jul 15 12:41:30.061100 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:30.061073 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:30.075440 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:30.075415 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdr67\" (UniqueName: \"kubernetes.io/projected/228efbb0-01e0-4d1a-b55f-035dbecc5dad-kube-api-access-pdr67\") pod \"228efbb0-01e0-4d1a-b55f-035dbecc5dad\" (UID: \"228efbb0-01e0-4d1a-b55f-035dbecc5dad\") " Jul 15 12:41:30.075560 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:30.075520 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/228efbb0-01e0-4d1a-b55f-035dbecc5dad-openshift-service-ca\") pod \"228efbb0-01e0-4d1a-b55f-035dbecc5dad\" (UID: \"228efbb0-01e0-4d1a-b55f-035dbecc5dad\") " Jul 15 12:41:30.075621 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:30.075577 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/228efbb0-01e0-4d1a-b55f-035dbecc5dad-maas-api-tls\") pod \"228efbb0-01e0-4d1a-b55f-035dbecc5dad\" (UID: \"228efbb0-01e0-4d1a-b55f-035dbecc5dad\") " Jul 15 12:41:30.075884 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:30.075858 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228efbb0-01e0-4d1a-b55f-035dbecc5dad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "228efbb0-01e0-4d1a-b55f-035dbecc5dad" (UID: "228efbb0-01e0-4d1a-b55f-035dbecc5dad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:41:30.079221 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:30.079193 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/228efbb0-01e0-4d1a-b55f-035dbecc5dad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "228efbb0-01e0-4d1a-b55f-035dbecc5dad" (UID: "228efbb0-01e0-4d1a-b55f-035dbecc5dad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:41:30.079221 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:30.079210 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228efbb0-01e0-4d1a-b55f-035dbecc5dad-kube-api-access-pdr67" (OuterVolumeSpecName: "kube-api-access-pdr67") pod "228efbb0-01e0-4d1a-b55f-035dbecc5dad" (UID: "228efbb0-01e0-4d1a-b55f-035dbecc5dad"). InnerVolumeSpecName "kube-api-access-pdr67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:41:30.176949 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:30.176871 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/228efbb0-01e0-4d1a-b55f-035dbecc5dad-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:41:30.176949 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:30.176896 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/228efbb0-01e0-4d1a-b55f-035dbecc5dad-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:41:30.176949 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:30.176905 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdr67\" (UniqueName: \"kubernetes.io/projected/228efbb0-01e0-4d1a-b55f-035dbecc5dad-kube-api-access-pdr67\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:41:30.319102 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:30.319072 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n"] Jul 15 12:41:30.326261 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:30.326232 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-af71886a-64fcb58685-55q9n"] Jul 15 12:41:30.853850 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:30.853817 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="228efbb0-01e0-4d1a-b55f-035dbecc5dad" path="/var/lib/kubelet/pods/228efbb0-01e0-4d1a-b55f-035dbecc5dad/volumes" Jul 15 12:41:35.520413 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:35.520374 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:35.621694 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:35.621660 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:35.739339 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:35.739310 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:35.835935 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:35.835861 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:36.848940 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:36.848904 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:36.944857 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:36.944819 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:37.088157 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.088113 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl"] Jul 15 12:41:37.088597 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.088577 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="942767bc-3c91-44a3-9033-dd0ebf58a324" containerName="revoke-keys" Jul 15 12:41:37.088645 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.088603 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="942767bc-3c91-44a3-9033-dd0ebf58a324" containerName="revoke-keys" Jul 15 12:41:37.088645 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.088617 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="942767bc-3c91-44a3-9033-dd0ebf58a324" containerName="revoke-keys" Jul 15 12:41:37.088645 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.088627 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="942767bc-3c91-44a3-9033-dd0ebf58a324" containerName="revoke-keys" Jul 15 12:41:37.088737 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.088645 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="942767bc-3c91-44a3-9033-dd0ebf58a324" containerName="revoke-keys" Jul 15 12:41:37.088737 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.088653 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="942767bc-3c91-44a3-9033-dd0ebf58a324" containerName="revoke-keys" Jul 15 12:41:37.088737 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.088667 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="228efbb0-01e0-4d1a-b55f-035dbecc5dad" containerName="maas-api" Jul 15 12:41:37.088737 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.088675 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="228efbb0-01e0-4d1a-b55f-035dbecc5dad" containerName="maas-api" Jul 15 12:41:37.088877 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.088756 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="942767bc-3c91-44a3-9033-dd0ebf58a324" containerName="revoke-keys" Jul 15 12:41:37.088877 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.088766 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="942767bc-3c91-44a3-9033-dd0ebf58a324" containerName="revoke-keys" Jul 15 12:41:37.088877 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.088797 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="228efbb0-01e0-4d1a-b55f-035dbecc5dad" containerName="maas-api" Jul 15 12:41:37.093733 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.093708 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" Jul 15 12:41:37.096590 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.096565 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f7995e41-serving-cert\"" Jul 15 12:41:37.096590 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.096589 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 12:41:37.097565 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.097546 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 12:41:37.097565 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.097557 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bnnf9\"" Jul 15 12:41:37.101655 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.101549 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl"] Jul 15 12:41:37.122429 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.122401 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:37.128626 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.128604 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdqbj\" (UniqueName: \"kubernetes.io/projected/8652d362-5117-4099-971f-3a4802582a2b-kube-api-access-zdqbj\") pod \"maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl\" (UID: \"8652d362-5117-4099-971f-3a4802582a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" Jul 15 12:41:37.128731 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.128633 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8652d362-5117-4099-971f-3a4802582a2b-maas-api-tls\") pod \"maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl\" (UID: \"8652d362-5117-4099-971f-3a4802582a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" Jul 15 12:41:37.128731 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.128664 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8652d362-5117-4099-971f-3a4802582a2b-openshift-service-ca\") pod \"maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl\" (UID: \"8652d362-5117-4099-971f-3a4802582a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" Jul 15 12:41:37.229473 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.229443 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8652d362-5117-4099-971f-3a4802582a2b-openshift-service-ca\") pod \"maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl\" (UID: \"8652d362-5117-4099-971f-3a4802582a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" Jul 15 12:41:37.229631 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.229498 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdqbj\" (UniqueName: \"kubernetes.io/projected/8652d362-5117-4099-971f-3a4802582a2b-kube-api-access-zdqbj\") pod \"maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl\" (UID: \"8652d362-5117-4099-971f-3a4802582a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" Jul 15 12:41:37.229631 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.229523 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8652d362-5117-4099-971f-3a4802582a2b-maas-api-tls\") pod \"maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl\" (UID: \"8652d362-5117-4099-971f-3a4802582a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" Jul 15 12:41:37.230089 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.230069 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8652d362-5117-4099-971f-3a4802582a2b-openshift-service-ca\") pod \"maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl\" (UID: \"8652d362-5117-4099-971f-3a4802582a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" Jul 15 12:41:37.232126 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.232106 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8652d362-5117-4099-971f-3a4802582a2b-maas-api-tls\") pod \"maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl\" (UID: \"8652d362-5117-4099-971f-3a4802582a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" Jul 15 12:41:37.239738 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.239720 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdqbj\" (UniqueName: \"kubernetes.io/projected/8652d362-5117-4099-971f-3a4802582a2b-kube-api-access-zdqbj\") pod \"maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl\" (UID: \"8652d362-5117-4099-971f-3a4802582a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" Jul 15 12:41:37.253994 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.253969 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:37.407050 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.406972 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" Jul 15 12:41:37.560742 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.560717 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl"] Jul 15 12:41:37.562545 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:41:37.562519 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8652d362_5117_4099_971f_3a4802582a2b.slice/crio-0dd9178e743c605ab8573a179cc0a4abccce2c0b770d163ae217d10e62635b00 WatchSource:0}: Error finding container 0dd9178e743c605ab8573a179cc0a4abccce2c0b770d163ae217d10e62635b00: Status 404 returned error can't find the container with id 0dd9178e743c605ab8573a179cc0a4abccce2c0b770d163ae217d10e62635b00 Jul 15 12:41:37.661473 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.661396 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:37.761912 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:37.761882 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:38.033930 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:38.033899 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" event={"ID":"8652d362-5117-4099-971f-3a4802582a2b","Type":"ContainerStarted","Data":"0dd9178e743c605ab8573a179cc0a4abccce2c0b770d163ae217d10e62635b00"} Jul 15 12:41:38.925133 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:38.925103 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:41.983388 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:41.983284 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r"] Jul 15 12:41:42.006794 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:41.983649 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="942767bc-3c91-44a3-9033-dd0ebf58a324" containerName="revoke-keys" Jul 15 12:41:42.006794 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:41.986631 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r" Jul 15 12:41:42.006794 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:41.989180 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kv8zr\"" Jul 15 12:41:42.006794 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:41.993432 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r"] Jul 15 12:41:42.052888 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:42.052855 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" event={"ID":"8652d362-5117-4099-971f-3a4802582a2b","Type":"ContainerStarted","Data":"8c707fe58b5ed199a57797ad5fc5d738a15a32d82e36331026a481f2ac2cc9d3"} Jul 15 12:41:42.053128 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:42.053108 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" Jul 15 12:41:42.065196 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:42.065169 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f7995e41-8md7r\" (UID: \"3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r" Jul 15 12:41:42.068147 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:42.068108 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" podStartSLOduration=1.372976387 podStartE2EDuration="5.068093161s" totalImagesPullingTime="3.695116774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:41:42.067368327 +0000 UTC m=+1825.701763276" watchObservedRunningTime="2026-07-15 12:41:42.068093161 +0000 UTC m=+1825.702488108" Jul 15 12:41:42.166018 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:42.165985 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f7995e41-8md7r\" (UID: \"3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r" Jul 15 12:41:42.166585 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:42.166565 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f7995e41-8md7r\" (UID: \"3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r" Jul 15 12:41:42.299556 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:42.299524 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r" Jul 15 12:41:42.425925 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:42.425899 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r"] Jul 15 12:41:42.428351 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:41:42.428327 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4c9dd5_93d5_4bbe_a32e_2b05b64a4d34.slice/crio-687631a76b5dcd91cb153e2e68d5b1d220ac092597df26ac13a40aad13a4b1ec WatchSource:0}: Error finding container 687631a76b5dcd91cb153e2e68d5b1d220ac092597df26ac13a40aad13a4b1ec: Status 404 returned error can't find the container with id 687631a76b5dcd91cb153e2e68d5b1d220ac092597df26ac13a40aad13a4b1ec Jul 15 12:41:43.058518 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:43.058483 2556 generic.go:356] "Generic (PLEG): container finished" podID="3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34" containerID="9f09b5e768ed45857709c2ba059914cb6689be887a90f055859601f6825b8ba0" exitCode=7 Jul 15 12:41:43.058877 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:43.058575 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r" event={"ID":"3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34","Type":"ContainerDied","Data":"9f09b5e768ed45857709c2ba059914cb6689be887a90f055859601f6825b8ba0"} Jul 15 12:41:43.058877 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:43.058629 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r" event={"ID":"3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34","Type":"ContainerStarted","Data":"687631a76b5dcd91cb153e2e68d5b1d220ac092597df26ac13a40aad13a4b1ec"} Jul 15 12:41:43.058877 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:43.058743 2556 scope.go:117] "RemoveContainer" containerID="9f09b5e768ed45857709c2ba059914cb6689be887a90f055859601f6825b8ba0" Jul 15 12:41:44.065038 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:44.065006 2556 generic.go:356] "Generic (PLEG): container finished" podID="3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34" containerID="15a7e8013584be5791ce081ba99d6785e4be88021e0158b491e76b383d9470a4" exitCode=7 Jul 15 12:41:44.065507 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:44.065088 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r" event={"ID":"3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34","Type":"ContainerDied","Data":"15a7e8013584be5791ce081ba99d6785e4be88021e0158b491e76b383d9470a4"} Jul 15 12:41:44.065507 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:44.065145 2556 scope.go:117] "RemoveContainer" containerID="9f09b5e768ed45857709c2ba059914cb6689be887a90f055859601f6825b8ba0" Jul 15 12:41:44.065507 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:44.065363 2556 scope.go:117] "RemoveContainer" containerID="15a7e8013584be5791ce081ba99d6785e4be88021e0158b491e76b383d9470a4" Jul 15 12:41:44.065659 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:41:44.065595 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f7995e41-8md7r_odh-ai-gateway-infra(3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r" podUID="3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34" Jul 15 12:41:45.071705 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:45.071679 2556 scope.go:117] "RemoveContainer" containerID="15a7e8013584be5791ce081ba99d6785e4be88021e0158b491e76b383d9470a4" Jul 15 12:41:45.072137 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:41:45.071866 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f7995e41-8md7r_odh-ai-gateway-infra(3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r" podUID="3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34" Jul 15 12:41:48.064062 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:48.064036 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" Jul 15 12:41:56.851148 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:56.851117 2556 scope.go:117] "RemoveContainer" containerID="15a7e8013584be5791ce081ba99d6785e4be88021e0158b491e76b383d9470a4" Jul 15 12:41:57.128827 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:57.128745 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r" event={"ID":"3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34","Type":"ContainerStarted","Data":"aae5bdb4a09a73245dfc8450eece27c69999ebce2b1887e857abb3b39850ea21"} Jul 15 12:41:57.144283 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:57.144240 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r" podStartSLOduration=16.14422535 podStartE2EDuration="16.14422535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:41:57.143204027 +0000 UTC m=+1840.777598976" watchObservedRunningTime="2026-07-15 12:41:57.14422535 +0000 UTC m=+1840.778620297" Jul 15 12:41:58.135436 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:58.135405 2556 generic.go:356] "Generic (PLEG): container finished" podID="3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34" containerID="aae5bdb4a09a73245dfc8450eece27c69999ebce2b1887e857abb3b39850ea21" exitCode=0 Jul 15 12:41:58.135835 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:58.135488 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r" event={"ID":"3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34","Type":"ContainerDied","Data":"aae5bdb4a09a73245dfc8450eece27c69999ebce2b1887e857abb3b39850ea21"} Jul 15 12:41:58.135835 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:58.135549 2556 scope.go:117] "RemoveContainer" containerID="15a7e8013584be5791ce081ba99d6785e4be88021e0158b491e76b383d9470a4" Jul 15 12:41:58.158049 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:58.158021 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r"] Jul 15 12:41:59.274395 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:59.274370 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r" Jul 15 12:41:59.285325 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:59.285308 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34-maas-api-service-ca\") pod \"3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34\" (UID: \"3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34\") " Jul 15 12:41:59.285599 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:59.285579 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34" (UID: "3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:41:59.386625 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:41:59.386586 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:42:00.146389 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:00.146358 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r" event={"ID":"3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34","Type":"ContainerDied","Data":"687631a76b5dcd91cb153e2e68d5b1d220ac092597df26ac13a40aad13a4b1ec"} Jul 15 12:42:00.146635 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:00.146396 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r" Jul 15 12:42:00.146635 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:00.146403 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="687631a76b5dcd91cb153e2e68d5b1d220ac092597df26ac13a40aad13a4b1ec" Jul 15 12:42:00.174325 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:00.174297 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r"] Jul 15 12:42:00.178490 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:00.178469 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-8md7r"] Jul 15 12:42:00.854654 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:00.854621 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34" path="/var/lib/kubelet/pods/3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34/volumes" Jul 15 12:42:32.021213 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:32.021179 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-7hx4k"] Jul 15 12:42:32.021657 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:32.021471 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34" containerName="revoke-keys" Jul 15 12:42:32.021657 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:32.021481 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34" containerName="revoke-keys" Jul 15 12:42:32.021657 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:32.021499 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34" containerName="revoke-keys" Jul 15 12:42:32.021657 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:32.021504 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34" containerName="revoke-keys" Jul 15 12:42:32.021657 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:32.021550 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34" containerName="revoke-keys" Jul 15 12:42:32.021657 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:32.021558 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34" containerName="revoke-keys" Jul 15 12:42:32.024373 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:32.024356 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-7hx4k" Jul 15 12:42:32.026697 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:32.026674 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kv8zr\"" Jul 15 12:42:32.031751 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:32.031341 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-7hx4k"] Jul 15 12:42:32.118605 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:32.118576 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b14a39-ac36-4203-853e-c1243e3381d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f7995e41-7hx4k\" (UID: \"96b14a39-ac36-4203-853e-c1243e3381d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-7hx4k" Jul 15 12:42:32.219815 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:32.219767 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b14a39-ac36-4203-853e-c1243e3381d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f7995e41-7hx4k\" (UID: \"96b14a39-ac36-4203-853e-c1243e3381d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-7hx4k" Jul 15 12:42:32.220411 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:32.220389 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b14a39-ac36-4203-853e-c1243e3381d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f7995e41-7hx4k\" (UID: \"96b14a39-ac36-4203-853e-c1243e3381d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-7hx4k" Jul 15 12:42:32.336124 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:32.336047 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-7hx4k" Jul 15 12:42:32.464623 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:32.464600 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-7hx4k"] Jul 15 12:42:32.466576 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:42:32.466547 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b14a39_ac36_4203_853e_c1243e3381d3.slice/crio-d7b37ec1babff7bfa7800335762846f2eacd6d898f942eec8803e6c3101ab7af WatchSource:0}: Error finding container d7b37ec1babff7bfa7800335762846f2eacd6d898f942eec8803e6c3101ab7af: Status 404 returned error can't find the container with id d7b37ec1babff7bfa7800335762846f2eacd6d898f942eec8803e6c3101ab7af Jul 15 12:42:33.307253 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:33.307218 2556 generic.go:356] "Generic (PLEG): container finished" podID="96b14a39-ac36-4203-853e-c1243e3381d3" containerID="75b01a6244703680d60cfb19b2fb7e2be1312f1e0e26e8924244e72eb3c73abb" exitCode=0 Jul 15 12:42:33.307615 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:33.307265 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-7hx4k" event={"ID":"96b14a39-ac36-4203-853e-c1243e3381d3","Type":"ContainerDied","Data":"75b01a6244703680d60cfb19b2fb7e2be1312f1e0e26e8924244e72eb3c73abb"} Jul 15 12:42:33.307615 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:33.307319 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-7hx4k" event={"ID":"96b14a39-ac36-4203-853e-c1243e3381d3","Type":"ContainerStarted","Data":"d7b37ec1babff7bfa7800335762846f2eacd6d898f942eec8803e6c3101ab7af"} Jul 15 12:42:34.446012 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:34.445990 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-7hx4k" Jul 15 12:42:34.538594 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:34.538549 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b14a39-ac36-4203-853e-c1243e3381d3-maas-api-service-ca\") pod \"96b14a39-ac36-4203-853e-c1243e3381d3\" (UID: \"96b14a39-ac36-4203-853e-c1243e3381d3\") " Jul 15 12:42:34.538942 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:34.538919 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b14a39-ac36-4203-853e-c1243e3381d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96b14a39-ac36-4203-853e-c1243e3381d3" (UID: "96b14a39-ac36-4203-853e-c1243e3381d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:42:34.639946 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:34.639877 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b14a39-ac36-4203-853e-c1243e3381d3-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:42:35.317578 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:35.317546 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-7hx4k" event={"ID":"96b14a39-ac36-4203-853e-c1243e3381d3","Type":"ContainerDied","Data":"d7b37ec1babff7bfa7800335762846f2eacd6d898f942eec8803e6c3101ab7af"} Jul 15 12:42:35.317816 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:35.317587 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7995e41-7hx4k" Jul 15 12:42:35.317816 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:35.317594 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7b37ec1babff7bfa7800335762846f2eacd6d898f942eec8803e6c3101ab7af" Jul 15 12:42:42.375909 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:42.375827 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl"] Jul 15 12:42:42.376330 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:42.376069 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" podUID="8652d362-5117-4099-971f-3a4802582a2b" containerName="maas-api" containerID="cri-o://8c707fe58b5ed199a57797ad5fc5d738a15a32d82e36331026a481f2ac2cc9d3" gracePeriod=30 Jul 15 12:42:42.403054 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:42.403014 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:42.624673 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:42.624649 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" Jul 15 12:42:42.702487 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:42.702406 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8652d362-5117-4099-971f-3a4802582a2b-maas-api-tls\") pod \"8652d362-5117-4099-971f-3a4802582a2b\" (UID: \"8652d362-5117-4099-971f-3a4802582a2b\") " Jul 15 12:42:42.702646 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:42.702500 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8652d362-5117-4099-971f-3a4802582a2b-openshift-service-ca\") pod \"8652d362-5117-4099-971f-3a4802582a2b\" (UID: \"8652d362-5117-4099-971f-3a4802582a2b\") " Jul 15 12:42:42.702646 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:42.702540 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdqbj\" (UniqueName: \"kubernetes.io/projected/8652d362-5117-4099-971f-3a4802582a2b-kube-api-access-zdqbj\") pod \"8652d362-5117-4099-971f-3a4802582a2b\" (UID: \"8652d362-5117-4099-971f-3a4802582a2b\") " Jul 15 12:42:42.702909 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:42.702887 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8652d362-5117-4099-971f-3a4802582a2b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8652d362-5117-4099-971f-3a4802582a2b" (UID: "8652d362-5117-4099-971f-3a4802582a2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:42:42.704617 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:42.704587 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8652d362-5117-4099-971f-3a4802582a2b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8652d362-5117-4099-971f-3a4802582a2b" (UID: "8652d362-5117-4099-971f-3a4802582a2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:42:42.704837 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:42.704811 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8652d362-5117-4099-971f-3a4802582a2b-kube-api-access-zdqbj" (OuterVolumeSpecName: "kube-api-access-zdqbj") pod "8652d362-5117-4099-971f-3a4802582a2b" (UID: "8652d362-5117-4099-971f-3a4802582a2b"). InnerVolumeSpecName "kube-api-access-zdqbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:42:42.803208 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:42.803174 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8652d362-5117-4099-971f-3a4802582a2b-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:42:42.803208 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:42.803204 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdqbj\" (UniqueName: \"kubernetes.io/projected/8652d362-5117-4099-971f-3a4802582a2b-kube-api-access-zdqbj\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:42:42.803208 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:42.803214 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8652d362-5117-4099-971f-3a4802582a2b-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:42:43.351581 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:43.351547 2556 generic.go:356] "Generic (PLEG): container finished" podID="8652d362-5117-4099-971f-3a4802582a2b" containerID="8c707fe58b5ed199a57797ad5fc5d738a15a32d82e36331026a481f2ac2cc9d3" exitCode=0 Jul 15 12:42:43.351797 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:43.351600 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" Jul 15 12:42:43.351797 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:43.351639 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" event={"ID":"8652d362-5117-4099-971f-3a4802582a2b","Type":"ContainerDied","Data":"8c707fe58b5ed199a57797ad5fc5d738a15a32d82e36331026a481f2ac2cc9d3"} Jul 15 12:42:43.351797 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:43.351690 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl" event={"ID":"8652d362-5117-4099-971f-3a4802582a2b","Type":"ContainerDied","Data":"0dd9178e743c605ab8573a179cc0a4abccce2c0b770d163ae217d10e62635b00"} Jul 15 12:42:43.351797 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:43.351713 2556 scope.go:117] "RemoveContainer" containerID="8c707fe58b5ed199a57797ad5fc5d738a15a32d82e36331026a481f2ac2cc9d3" Jul 15 12:42:43.360303 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:43.360287 2556 scope.go:117] "RemoveContainer" containerID="8c707fe58b5ed199a57797ad5fc5d738a15a32d82e36331026a481f2ac2cc9d3" Jul 15 12:42:43.360519 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:42:43.360498 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c707fe58b5ed199a57797ad5fc5d738a15a32d82e36331026a481f2ac2cc9d3\": container with ID starting with 8c707fe58b5ed199a57797ad5fc5d738a15a32d82e36331026a481f2ac2cc9d3 not found: ID does not exist" containerID="8c707fe58b5ed199a57797ad5fc5d738a15a32d82e36331026a481f2ac2cc9d3" Jul 15 12:42:43.360571 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:43.360531 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c707fe58b5ed199a57797ad5fc5d738a15a32d82e36331026a481f2ac2cc9d3"} err="failed to get container status \"8c707fe58b5ed199a57797ad5fc5d738a15a32d82e36331026a481f2ac2cc9d3\": rpc error: code = NotFound desc = could not find container \"8c707fe58b5ed199a57797ad5fc5d738a15a32d82e36331026a481f2ac2cc9d3\": container with ID starting with 8c707fe58b5ed199a57797ad5fc5d738a15a32d82e36331026a481f2ac2cc9d3 not found: ID does not exist" Jul 15 12:42:43.368082 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:43.368047 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl"] Jul 15 12:42:43.371567 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:43.371547 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7995e41-9979b5c7b-l9zfl"] Jul 15 12:42:44.853699 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:44.853666 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8652d362-5117-4099-971f-3a4802582a2b" path="/var/lib/kubelet/pods/8652d362-5117-4099-971f-3a4802582a2b/volumes" Jul 15 12:42:47.098468 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:47.098432 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:52.811318 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:52.811265 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:52.872546 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:52.872511 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:56.571395 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:56.571364 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:58.160765 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.160732 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:58.428306 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.428216 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52"] Jul 15 12:42:58.428712 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.428688 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34" containerName="revoke-keys" Jul 15 12:42:58.428712 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.428713 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34" containerName="revoke-keys" Jul 15 12:42:58.428920 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.428734 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8652d362-5117-4099-971f-3a4802582a2b" containerName="maas-api" Jul 15 12:42:58.428920 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.428742 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8652d362-5117-4099-971f-3a4802582a2b" containerName="maas-api" Jul 15 12:42:58.428920 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.428767 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96b14a39-ac36-4203-853e-c1243e3381d3" containerName="revoke-keys" Jul 15 12:42:58.428920 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.428798 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="96b14a39-ac36-4203-853e-c1243e3381d3" containerName="revoke-keys" Jul 15 12:42:58.428920 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.428876 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b4c9dd5-93d5-4bbe-a32e-2b05b64a4d34" containerName="revoke-keys" Jul 15 12:42:58.428920 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.428891 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8652d362-5117-4099-971f-3a4802582a2b" containerName="maas-api" Jul 15 12:42:58.428920 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.428900 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="96b14a39-ac36-4203-853e-c1243e3381d3" containerName="revoke-keys" Jul 15 12:42:58.431724 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.431705 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.434258 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.434233 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-34aa8850-openshift-default-dockercfg-fgm99\"" Jul 15 12:42:58.445827 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.444059 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52"] Jul 15 12:42:58.467715 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.467685 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:58.540064 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.540011 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-credential-socket\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.540285 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.540192 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-workload-certs\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.540285 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.540265 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-workload-socket\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.540826 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.540498 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-envoy\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.540946 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.540882 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-podinfo\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.540946 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.540916 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-token\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.541036 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.540999 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-data\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.541101 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.541080 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istiod-ca-cert\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.541159 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.541123 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr67s\" (UniqueName: \"kubernetes.io/projected/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-kube-api-access-tr67s\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.641542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.641507 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-podinfo\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.641542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.641540 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-token\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.641820 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.641565 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-data\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.641820 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.641582 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istiod-ca-cert\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.641820 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.641599 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr67s\" (UniqueName: \"kubernetes.io/projected/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-kube-api-access-tr67s\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.641820 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.641744 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-credential-socket\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.642007 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.641855 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-workload-certs\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.642007 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.641922 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-workload-socket\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.642007 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.641960 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-envoy\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.642210 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.642101 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-data\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.642210 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.642188 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-credential-socket\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.642304 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.642251 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-workload-certs\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.642304 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.642292 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-workload-socket\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.642417 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.642397 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istiod-ca-cert\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.644158 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.644135 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-envoy\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.644282 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.644172 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-podinfo\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.649282 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.649256 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-token\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.649421 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.649405 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr67s\" (UniqueName: \"kubernetes.io/projected/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-kube-api-access-tr67s\") pod \"e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.744298 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.744223 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:42:58.878700 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:58.878670 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52"] Jul 15 12:42:58.881140 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:42:58.881092 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea5214c_bbf0_46c4_9ec6_1d76c56d057e.slice/crio-112757a48e599bd52aab94681f73c744a0d95a6cebc69a8b794bf2de242f1a38 WatchSource:0}: Error finding container 112757a48e599bd52aab94681f73c744a0d95a6cebc69a8b794bf2de242f1a38: Status 404 returned error can't find the container with id 112757a48e599bd52aab94681f73c744a0d95a6cebc69a8b794bf2de242f1a38 Jul 15 12:42:59.420634 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:59.420599 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" event={"ID":"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e","Type":"ContainerStarted","Data":"112757a48e599bd52aab94681f73c744a0d95a6cebc69a8b794bf2de242f1a38"} Jul 15 12:42:59.598743 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:59.598703 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:59.714313 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:59.714226 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h"] Jul 15 12:42:59.717542 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:59.717523 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" Jul 15 12:42:59.721269 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:59.721246 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bnnf9\"" Jul 15 12:42:59.721653 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:59.721639 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 12:42:59.722369 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:59.722345 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-34aa8850-serving-cert\"" Jul 15 12:42:59.723401 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:59.723383 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 12:42:59.735505 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:59.735483 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h"] Jul 15 12:42:59.736555 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:59.736528 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:59.821524 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:59.821493 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:59.852596 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:59.852555 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/714b12e0-2df2-4a12-84d7-1c1d97e1df53-maas-api-tls\") pod \"maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h\" (UID: \"714b12e0-2df2-4a12-84d7-1c1d97e1df53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" Jul 15 12:42:59.852931 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:59.852617 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4x8l\" (UniqueName: \"kubernetes.io/projected/714b12e0-2df2-4a12-84d7-1c1d97e1df53-kube-api-access-n4x8l\") pod \"maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h\" (UID: \"714b12e0-2df2-4a12-84d7-1c1d97e1df53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" Jul 15 12:42:59.852931 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:59.852648 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/714b12e0-2df2-4a12-84d7-1c1d97e1df53-openshift-service-ca\") pod \"maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h\" (UID: \"714b12e0-2df2-4a12-84d7-1c1d97e1df53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" Jul 15 12:42:59.953382 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:59.953334 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/714b12e0-2df2-4a12-84d7-1c1d97e1df53-maas-api-tls\") pod \"maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h\" (UID: \"714b12e0-2df2-4a12-84d7-1c1d97e1df53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" Jul 15 12:42:59.953382 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:59.953383 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4x8l\" (UniqueName: \"kubernetes.io/projected/714b12e0-2df2-4a12-84d7-1c1d97e1df53-kube-api-access-n4x8l\") pod \"maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h\" (UID: \"714b12e0-2df2-4a12-84d7-1c1d97e1df53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" Jul 15 12:42:59.953605 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:59.953407 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/714b12e0-2df2-4a12-84d7-1c1d97e1df53-openshift-service-ca\") pod \"maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h\" (UID: \"714b12e0-2df2-4a12-84d7-1c1d97e1df53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" Jul 15 12:42:59.954091 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:59.954047 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/714b12e0-2df2-4a12-84d7-1c1d97e1df53-openshift-service-ca\") pod \"maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h\" (UID: \"714b12e0-2df2-4a12-84d7-1c1d97e1df53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" Jul 15 12:42:59.956248 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:59.956225 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/714b12e0-2df2-4a12-84d7-1c1d97e1df53-maas-api-tls\") pod \"maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h\" (UID: \"714b12e0-2df2-4a12-84d7-1c1d97e1df53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" Jul 15 12:42:59.962257 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:42:59.962204 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4x8l\" (UniqueName: \"kubernetes.io/projected/714b12e0-2df2-4a12-84d7-1c1d97e1df53-kube-api-access-n4x8l\") pod \"maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h\" (UID: \"714b12e0-2df2-4a12-84d7-1c1d97e1df53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" Jul 15 12:43:00.030014 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:00.029951 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" Jul 15 12:43:00.187009 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:00.186978 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h"] Jul 15 12:43:00.189578 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:43:00.189380 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714b12e0_2df2_4a12_84d7_1c1d97e1df53.slice/crio-b11eb888c04f44dabfc123f7e2fa836c47ad2d37c1ed74b111d1525b16eef57c WatchSource:0}: Error finding container b11eb888c04f44dabfc123f7e2fa836c47ad2d37c1ed74b111d1525b16eef57c: Status 404 returned error can't find the container with id b11eb888c04f44dabfc123f7e2fa836c47ad2d37c1ed74b111d1525b16eef57c Jul 15 12:43:00.318038 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:00.317962 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:00.420019 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:00.419981 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:00.427076 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:00.427042 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" event={"ID":"714b12e0-2df2-4a12-84d7-1c1d97e1df53","Type":"ContainerStarted","Data":"b11eb888c04f44dabfc123f7e2fa836c47ad2d37c1ed74b111d1525b16eef57c"} Jul 15 12:43:01.410642 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:01.410608 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:43:01.410712 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:01.410679 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:43:01.410751 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:01.410713 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:43:02.440565 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:02.440526 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" event={"ID":"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e","Type":"ContainerStarted","Data":"728983af0b9852769fec4573583d98a7551e3d642f12dfc0e1b733d4c7818f78"} Jul 15 12:43:02.442036 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:02.442016 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" event={"ID":"714b12e0-2df2-4a12-84d7-1c1d97e1df53","Type":"ContainerStarted","Data":"cc9a20e58a56d6f24fe9f9205b4b330a1c23f4d9e36f4cc35b208fdfea73bef6"} Jul 15 12:43:02.465124 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:02.465073 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" podStartSLOduration=1.9378575489999998 podStartE2EDuration="4.465055449s" totalImagesPullingTime="2.5271979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:43:02.464609571 +0000 UTC m=+1906.099004520" watchObservedRunningTime="2026-07-15 12:43:02.465055449 +0000 UTC m=+1906.099450398" Jul 15 12:43:02.482940 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:02.482900 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" podStartSLOduration=2.264075392 podStartE2EDuration="3.482886157s" totalImagesPullingTime="1.218810765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:43:02.482160281 +0000 UTC m=+1906.116555231" watchObservedRunningTime="2026-07-15 12:43:02.482886157 +0000 UTC m=+1906.117281145" Jul 15 12:43:02.745029 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:02.744939 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:43:02.749647 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:02.749623 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:43:03.447537 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:03.447506 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:43:03.448043 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:03.447563 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" Jul 15 12:43:03.448661 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:03.448643 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:43:05.448102 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:05.448059 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:05.553189 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:05.553120 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:08.617355 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:08.617310 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:09.456689 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:09.456658 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" Jul 15 12:43:13.495718 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:13.495683 2556 generic.go:356] "Generic (PLEG): container finished" podID="6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" containerID="728983af0b9852769fec4573583d98a7551e3d642f12dfc0e1b733d4c7818f78" exitCode=0 Jul 15 12:43:13.496202 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:13.495756 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" event={"ID":"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e","Type":"ContainerDied","Data":"728983af0b9852769fec4573583d98a7551e3d642f12dfc0e1b733d4c7818f78"} Jul 15 12:43:13.496271 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:13.496250 2556 scope.go:117] "RemoveContainer" containerID="728983af0b9852769fec4573583d98a7551e3d642f12dfc0e1b733d4c7818f78" Jul 15 12:43:13.498620 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:13.498588 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:43:13.498752 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:13.498677 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:43:13.498752 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:13.498708 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:43:14.507215 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:14.507165 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" event={"ID":"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e","Type":"ContainerStarted","Data":"b6411c8bb75570e538d8ca11a839791d81f14eee69e24b2b87a3f759c66250e7"} Jul 15 12:43:14.744664 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:14.744625 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:43:14.745855 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:14.745822 2556 patch_prober.go:28] interesting pod/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 15 12:43:14.745971 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:14.745880 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" podUID="6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 15 12:43:15.744980 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:15.744936 2556 patch_prober.go:28] interesting pod/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 15 12:43:15.745362 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:15.745012 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" podUID="6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 15 12:43:16.745549 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:16.745512 2556 patch_prober.go:28] interesting pod/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 15 12:43:16.745963 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:16.745588 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" podUID="6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 15 12:43:17.744731 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:17.744693 2556 patch_prober.go:28] interesting pod/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 15 12:43:17.744979 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:17.744752 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" podUID="6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 15 12:43:18.423961 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:18.423921 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:18.447679 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:18.447653 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:43:18.524874 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:18.524827 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:18.749498 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:18.749471 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:43:18.750465 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:18.750450 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:43:21.035421 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:21.035386 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:21.136284 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:43:21.136251 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:04.509632 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:04.509598 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h"] Jul 15 12:44:04.510573 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:04.510538 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" podUID="714b12e0-2df2-4a12-84d7-1c1d97e1df53" containerName="maas-api" containerID="cri-o://cc9a20e58a56d6f24fe9f9205b4b330a1c23f4d9e36f4cc35b208fdfea73bef6" gracePeriod=30 Jul 15 12:44:04.533865 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:04.533840 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:04.734944 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:04.734913 2556 generic.go:356] "Generic (PLEG): container finished" podID="714b12e0-2df2-4a12-84d7-1c1d97e1df53" containerID="cc9a20e58a56d6f24fe9f9205b4b330a1c23f4d9e36f4cc35b208fdfea73bef6" exitCode=0 Jul 15 12:44:04.735094 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:04.734946 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" event={"ID":"714b12e0-2df2-4a12-84d7-1c1d97e1df53","Type":"ContainerDied","Data":"cc9a20e58a56d6f24fe9f9205b4b330a1c23f4d9e36f4cc35b208fdfea73bef6"} Jul 15 12:44:04.770454 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:04.770400 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" Jul 15 12:44:04.899263 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:04.899229 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4x8l\" (UniqueName: \"kubernetes.io/projected/714b12e0-2df2-4a12-84d7-1c1d97e1df53-kube-api-access-n4x8l\") pod \"714b12e0-2df2-4a12-84d7-1c1d97e1df53\" (UID: \"714b12e0-2df2-4a12-84d7-1c1d97e1df53\") " Jul 15 12:44:04.899411 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:04.899305 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/714b12e0-2df2-4a12-84d7-1c1d97e1df53-maas-api-tls\") pod \"714b12e0-2df2-4a12-84d7-1c1d97e1df53\" (UID: \"714b12e0-2df2-4a12-84d7-1c1d97e1df53\") " Jul 15 12:44:04.899411 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:04.899332 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/714b12e0-2df2-4a12-84d7-1c1d97e1df53-openshift-service-ca\") pod \"714b12e0-2df2-4a12-84d7-1c1d97e1df53\" (UID: \"714b12e0-2df2-4a12-84d7-1c1d97e1df53\") " Jul 15 12:44:04.899816 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:04.899754 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714b12e0-2df2-4a12-84d7-1c1d97e1df53-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "714b12e0-2df2-4a12-84d7-1c1d97e1df53" (UID: "714b12e0-2df2-4a12-84d7-1c1d97e1df53"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:44:04.901515 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:04.901488 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714b12e0-2df2-4a12-84d7-1c1d97e1df53-kube-api-access-n4x8l" (OuterVolumeSpecName: "kube-api-access-n4x8l") pod "714b12e0-2df2-4a12-84d7-1c1d97e1df53" (UID: "714b12e0-2df2-4a12-84d7-1c1d97e1df53"). InnerVolumeSpecName "kube-api-access-n4x8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:44:04.901597 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:04.901504 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/714b12e0-2df2-4a12-84d7-1c1d97e1df53-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "714b12e0-2df2-4a12-84d7-1c1d97e1df53" (UID: "714b12e0-2df2-4a12-84d7-1c1d97e1df53"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:44:05.000666 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:05.000635 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/714b12e0-2df2-4a12-84d7-1c1d97e1df53-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:44:05.000666 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:05.000663 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/714b12e0-2df2-4a12-84d7-1c1d97e1df53-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:44:05.000666 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:05.000673 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4x8l\" (UniqueName: \"kubernetes.io/projected/714b12e0-2df2-4a12-84d7-1c1d97e1df53-kube-api-access-n4x8l\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:44:05.741789 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:05.741740 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" Jul 15 12:44:05.742213 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:05.741785 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h" event={"ID":"714b12e0-2df2-4a12-84d7-1c1d97e1df53","Type":"ContainerDied","Data":"b11eb888c04f44dabfc123f7e2fa836c47ad2d37c1ed74b111d1525b16eef57c"} Jul 15 12:44:05.742213 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:05.741850 2556 scope.go:117] "RemoveContainer" containerID="cc9a20e58a56d6f24fe9f9205b4b330a1c23f4d9e36f4cc35b208fdfea73bef6" Jul 15 12:44:05.762866 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:05.762840 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h"] Jul 15 12:44:05.767190 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:05.767167 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-34aa8850-ddd7fb5c7-kbg6h"] Jul 15 12:44:06.854199 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:06.854158 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="714b12e0-2df2-4a12-84d7-1c1d97e1df53" path="/var/lib/kubelet/pods/714b12e0-2df2-4a12-84d7-1c1d97e1df53/volumes" Jul 15 12:44:09.097924 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:09.097828 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:15.137298 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:15.137247 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:15.237407 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:15.237367 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:19.267382 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:19.267341 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52"] Jul 15 12:44:19.267815 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:19.267593 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" podUID="6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" containerName="istio-proxy" containerID="cri-o://b6411c8bb75570e538d8ca11a839791d81f14eee69e24b2b87a3f759c66250e7" gracePeriod=30 Jul 15 12:44:19.290329 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:19.290295 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:19.891337 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:19.891294 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:19.993678 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:19.993648 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:20.114284 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:20.114250 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:21.508743 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:21.508708 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:24.520808 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.518499 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:44:24.565549 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.565467 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-envoy\") pod \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " Jul 15 12:44:24.565704 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.565551 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tr67s\" (UniqueName: \"kubernetes.io/projected/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-kube-api-access-tr67s\") pod \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " Jul 15 12:44:24.565704 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.565585 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-workload-certs\") pod \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " Jul 15 12:44:24.565704 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.565662 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-credential-socket\") pod \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " Jul 15 12:44:24.565704 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.565686 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-workload-socket\") pod \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " Jul 15 12:44:24.565945 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.565710 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-podinfo\") pod \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " Jul 15 12:44:24.565945 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.565758 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-token\") pod \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " Jul 15 12:44:24.565945 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.565809 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istiod-ca-cert\") pod \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " Jul 15 12:44:24.565945 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.565848 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-data\") pod \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\" (UID: \"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e\") " Jul 15 12:44:24.565945 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.565910 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" (UID: "6ea5214c-bbf0-46c4-9ec6-1d76c56d057e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:44:24.566235 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.565977 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" (UID: "6ea5214c-bbf0-46c4-9ec6-1d76c56d057e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:44:24.566235 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.566109 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-credential-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:44:24.566235 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.566131 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-workload-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:44:24.566628 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.566603 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" (UID: "6ea5214c-bbf0-46c4-9ec6-1d76c56d057e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:44:24.566628 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.566609 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" (UID: "6ea5214c-bbf0-46c4-9ec6-1d76c56d057e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:44:24.567046 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.567018 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-data" (OuterVolumeSpecName: "istio-data") pod "6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" (UID: "6ea5214c-bbf0-46c4-9ec6-1d76c56d057e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:44:24.568669 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.568648 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" (UID: "6ea5214c-bbf0-46c4-9ec6-1d76c56d057e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:44:24.568801 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.568753 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-token" (OuterVolumeSpecName: "istio-token") pod "6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" (UID: "6ea5214c-bbf0-46c4-9ec6-1d76c56d057e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:44:24.569055 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.569032 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" (UID: "6ea5214c-bbf0-46c4-9ec6-1d76c56d057e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:44:24.569252 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.569234 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-kube-api-access-tr67s" (OuterVolumeSpecName: "kube-api-access-tr67s") pod "6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" (UID: "6ea5214c-bbf0-46c4-9ec6-1d76c56d057e"). InnerVolumeSpecName "kube-api-access-tr67s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:44:24.667117 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.667080 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:44:24.667117 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.667109 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istiod-ca-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:44:24.667117 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.667119 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-data\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:44:24.667117 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.667127 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-envoy\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:44:24.667363 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.667135 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr67s\" (UniqueName: \"kubernetes.io/projected/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-kube-api-access-tr67s\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:44:24.667363 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.667145 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-workload-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:44:24.667363 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.667154 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e-istio-podinfo\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:44:24.834526 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.834440 2556 generic.go:356] "Generic (PLEG): container finished" podID="6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" containerID="b6411c8bb75570e538d8ca11a839791d81f14eee69e24b2b87a3f759c66250e7" exitCode=0 Jul 15 12:44:24.834526 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.834490 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" Jul 15 12:44:24.834696 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.834528 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" event={"ID":"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e","Type":"ContainerDied","Data":"b6411c8bb75570e538d8ca11a839791d81f14eee69e24b2b87a3f759c66250e7"} Jul 15 12:44:24.834696 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.834606 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52" event={"ID":"6ea5214c-bbf0-46c4-9ec6-1d76c56d057e","Type":"ContainerDied","Data":"112757a48e599bd52aab94681f73c744a0d95a6cebc69a8b794bf2de242f1a38"} Jul 15 12:44:24.834696 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.834628 2556 scope.go:117] "RemoveContainer" containerID="b6411c8bb75570e538d8ca11a839791d81f14eee69e24b2b87a3f759c66250e7" Jul 15 12:44:24.843610 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.843594 2556 scope.go:117] "RemoveContainer" containerID="728983af0b9852769fec4573583d98a7551e3d642f12dfc0e1b733d4c7818f78" Jul 15 12:44:24.851543 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.851522 2556 scope.go:117] "RemoveContainer" containerID="b6411c8bb75570e538d8ca11a839791d81f14eee69e24b2b87a3f759c66250e7" Jul 15 12:44:24.851805 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:44:24.851782 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6411c8bb75570e538d8ca11a839791d81f14eee69e24b2b87a3f759c66250e7\": container with ID starting with b6411c8bb75570e538d8ca11a839791d81f14eee69e24b2b87a3f759c66250e7 not found: ID does not exist" containerID="b6411c8bb75570e538d8ca11a839791d81f14eee69e24b2b87a3f759c66250e7" Jul 15 12:44:24.851906 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.851811 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6411c8bb75570e538d8ca11a839791d81f14eee69e24b2b87a3f759c66250e7"} err="failed to get container status \"b6411c8bb75570e538d8ca11a839791d81f14eee69e24b2b87a3f759c66250e7\": rpc error: code = NotFound desc = could not find container \"b6411c8bb75570e538d8ca11a839791d81f14eee69e24b2b87a3f759c66250e7\": container with ID starting with b6411c8bb75570e538d8ca11a839791d81f14eee69e24b2b87a3f759c66250e7 not found: ID does not exist" Jul 15 12:44:24.851906 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.851829 2556 scope.go:117] "RemoveContainer" containerID="728983af0b9852769fec4573583d98a7551e3d642f12dfc0e1b733d4c7818f78" Jul 15 12:44:24.852078 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:44:24.852060 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"728983af0b9852769fec4573583d98a7551e3d642f12dfc0e1b733d4c7818f78\": container with ID starting with 728983af0b9852769fec4573583d98a7551e3d642f12dfc0e1b733d4c7818f78 not found: ID does not exist" containerID="728983af0b9852769fec4573583d98a7551e3d642f12dfc0e1b733d4c7818f78" Jul 15 12:44:24.852116 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.852084 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728983af0b9852769fec4573583d98a7551e3d642f12dfc0e1b733d4c7818f78"} err="failed to get container status \"728983af0b9852769fec4573583d98a7551e3d642f12dfc0e1b733d4c7818f78\": rpc error: code = NotFound desc = could not find container \"728983af0b9852769fec4573583d98a7551e3d642f12dfc0e1b733d4c7818f78\": container with ID starting with 728983af0b9852769fec4573583d98a7551e3d642f12dfc0e1b733d4c7818f78 not found: ID does not exist" Jul 15 12:44:24.855061 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.855042 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52"] Jul 15 12:44:24.858540 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:24.858522 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-34aa8850-openshift-default-5d46fb66c9-7xb52"] Jul 15 12:44:25.846209 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:25.846180 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:26.853530 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:26.853489 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" path="/var/lib/kubelet/pods/6ea5214c-bbf0-46c4-9ec6-1d76c56d057e/volumes" Jul 15 12:44:27.092820 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:27.092787 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:27.197613 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:27.197539 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:27.292991 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:27.292956 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:27.413203 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:27.413169 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:27.865989 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:27.865959 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:53.880816 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:53.880754 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:59.550162 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:59.550114 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:59.657176 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:59.657138 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:59.752323 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:44:59.752295 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:15.650063 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:15.650031 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:21.352034 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:21.351988 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:22.798014 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:22.797980 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:22.898990 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:22.898957 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:32.107605 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:32.107567 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:32.942413 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:32.942374 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:37.920560 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:37.920470 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:38.537873 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:38.537835 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:38.648889 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:38.648857 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:38.788404 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:38.788316 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:40.618822 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:40.618768 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:40.718486 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:40.718457 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:41.457356 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:41.457313 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:43.312870 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:43.312833 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:47.154872 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:47.154844 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:47.218754 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:47.218722 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:47.845790 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:47.845747 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:53.394146 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.394104 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:53.494538 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.494499 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:53.630719 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.630686 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:53.909050 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.908981 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc"] Jul 15 12:45:53.909311 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.909296 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="714b12e0-2df2-4a12-84d7-1c1d97e1df53" containerName="maas-api" Jul 15 12:45:53.909311 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.909311 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="714b12e0-2df2-4a12-84d7-1c1d97e1df53" containerName="maas-api" Jul 15 12:45:53.909433 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.909328 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" containerName="istio-proxy" Jul 15 12:45:53.909433 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.909334 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" containerName="istio-proxy" Jul 15 12:45:53.909433 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.909343 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" containerName="istio-proxy" Jul 15 12:45:53.909433 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.909349 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" containerName="istio-proxy" Jul 15 12:45:53.909433 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.909392 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" containerName="istio-proxy" Jul 15 12:45:53.909433 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.909400 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="714b12e0-2df2-4a12-84d7-1c1d97e1df53" containerName="maas-api" Jul 15 12:45:53.909433 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.909406 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ea5214c-bbf0-46c4-9ec6-1d76c56d057e" containerName="istio-proxy" Jul 15 12:45:53.912287 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.912268 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:53.914857 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.914840 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c7c0f6bf-openshift-default-dockercfg-zmqrn\"" Jul 15 12:45:53.927617 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.927589 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc"] Jul 15 12:45:53.932675 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.932651 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:53.975890 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.975863 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-envoy\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:53.975890 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.975893 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-token\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:53.976085 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.975923 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hnkb\" (UniqueName: \"kubernetes.io/projected/385dd7f4-e237-4b56-bdaa-e25244a624de-kube-api-access-9hnkb\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:53.976085 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.975962 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-workload-certs\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:53.976085 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.975997 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-podinfo\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:53.976085 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.976018 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385dd7f4-e237-4b56-bdaa-e25244a624de-istiod-ca-cert\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:53.976235 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.976104 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-credential-socket\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:53.976235 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.976148 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-workload-socket\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:53.976235 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:53.976167 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-data\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:54.077362 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.077325 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-podinfo\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:54.077362 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.077362 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385dd7f4-e237-4b56-bdaa-e25244a624de-istiod-ca-cert\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:54.077613 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.077389 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-credential-socket\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:54.077613 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.077417 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-workload-socket\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:54.077613 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.077435 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-data\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:54.077613 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.077453 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-envoy\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:54.077613 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.077471 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-token\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:54.077613 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.077500 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hnkb\" (UniqueName: \"kubernetes.io/projected/385dd7f4-e237-4b56-bdaa-e25244a624de-kube-api-access-9hnkb\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:54.077613 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.077528 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-workload-certs\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:54.077989 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.077850 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-credential-socket\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:54.077989 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.077972 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-data\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:54.078130 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.078111 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-workload-certs\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:54.078310 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.078134 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-workload-socket\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:54.078310 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.078250 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385dd7f4-e237-4b56-bdaa-e25244a624de-istiod-ca-cert\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:54.079937 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.079915 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-envoy\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:54.080158 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.080138 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-podinfo\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:54.096525 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.096505 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-token\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:54.096630 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.096618 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hnkb\" (UniqueName: \"kubernetes.io/projected/385dd7f4-e237-4b56-bdaa-e25244a624de-kube-api-access-9hnkb\") pod \"e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:54.228374 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.228283 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:54.365856 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.365822 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc"] Jul 15 12:45:54.369274 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:45:54.369245 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385dd7f4_e237_4b56_bdaa_e25244a624de.slice/crio-c68e6a4e91791414982075c9a09cbfcc9700422450ecd6b3cd6fe157dd1506b9 WatchSource:0}: Error finding container c68e6a4e91791414982075c9a09cbfcc9700422450ecd6b3cd6fe157dd1506b9: Status 404 returned error can't find the container with id c68e6a4e91791414982075c9a09cbfcc9700422450ecd6b3cd6fe157dd1506b9 Jul 15 12:45:54.371325 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.371291 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:45:54.371382 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.371361 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:45:54.371429 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.371390 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:45:54.595575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:54.595533 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:55.230204 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:55.230168 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" event={"ID":"385dd7f4-e237-4b56-bdaa-e25244a624de","Type":"ContainerStarted","Data":"ea104a73a1a7156481da63c2f1461d38e09c4534b6d4963d101db1a75541e249"} Jul 15 12:45:55.230409 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:55.230224 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" event={"ID":"385dd7f4-e237-4b56-bdaa-e25244a624de","Type":"ContainerStarted","Data":"c68e6a4e91791414982075c9a09cbfcc9700422450ecd6b3cd6fe157dd1506b9"} Jul 15 12:45:55.253384 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:55.253314 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" podStartSLOduration=2.253292332 podStartE2EDuration="2.253292332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:45:55.250069979 +0000 UTC m=+2078.884464928" watchObservedRunningTime="2026-07-15 12:45:55.253292332 +0000 UTC m=+2078.887687720" Jul 15 12:45:56.229092 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:56.229059 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:56.234234 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:56.234210 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:56.234826 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:56.234803 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:56.235604 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:56.235587 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:45:57.149023 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.148990 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:57.224966 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.224933 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:57.230042 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.230017 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc"] Jul 15 12:45:57.233282 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.233263 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" Jul 15 12:45:57.236255 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.236232 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c7c0f6bf-serving-cert\"" Jul 15 12:45:57.236490 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.236475 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bnnf9\"" Jul 15 12:45:57.237656 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.237637 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 12:45:57.237786 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.237672 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 12:45:57.244554 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.244525 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc"] Jul 15 12:45:57.299229 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.299200 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:57.309893 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.309873 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb75fce-3be3-4019-b39c-3243211a06b7-maas-api-tls\") pod \"maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc\" (UID: \"fcb75fce-3be3-4019-b39c-3243211a06b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" Jul 15 12:45:57.310072 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.310048 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r9xg\" (UniqueName: \"kubernetes.io/projected/fcb75fce-3be3-4019-b39c-3243211a06b7-kube-api-access-6r9xg\") pod \"maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc\" (UID: \"fcb75fce-3be3-4019-b39c-3243211a06b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" Jul 15 12:45:57.310170 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.310151 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb75fce-3be3-4019-b39c-3243211a06b7-openshift-service-ca\") pod \"maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc\" (UID: \"fcb75fce-3be3-4019-b39c-3243211a06b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" Jul 15 12:45:57.400860 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.400743 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:57.410803 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.410764 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb75fce-3be3-4019-b39c-3243211a06b7-maas-api-tls\") pod \"maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc\" (UID: \"fcb75fce-3be3-4019-b39c-3243211a06b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" Jul 15 12:45:57.410942 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.410846 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r9xg\" (UniqueName: \"kubernetes.io/projected/fcb75fce-3be3-4019-b39c-3243211a06b7-kube-api-access-6r9xg\") pod \"maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc\" (UID: \"fcb75fce-3be3-4019-b39c-3243211a06b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" Jul 15 12:45:57.410942 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.410876 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb75fce-3be3-4019-b39c-3243211a06b7-openshift-service-ca\") pod \"maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc\" (UID: \"fcb75fce-3be3-4019-b39c-3243211a06b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" Jul 15 12:45:57.411482 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.411461 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb75fce-3be3-4019-b39c-3243211a06b7-openshift-service-ca\") pod \"maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc\" (UID: \"fcb75fce-3be3-4019-b39c-3243211a06b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" Jul 15 12:45:57.413529 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.413505 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb75fce-3be3-4019-b39c-3243211a06b7-maas-api-tls\") pod \"maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc\" (UID: \"fcb75fce-3be3-4019-b39c-3243211a06b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" Jul 15 12:45:57.425250 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.425223 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r9xg\" (UniqueName: \"kubernetes.io/projected/fcb75fce-3be3-4019-b39c-3243211a06b7-kube-api-access-6r9xg\") pod \"maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc\" (UID: \"fcb75fce-3be3-4019-b39c-3243211a06b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" Jul 15 12:45:57.547363 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.547336 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" Jul 15 12:45:57.687256 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.687175 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc"] Jul 15 12:45:57.689401 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:45:57.689371 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb75fce_3be3_4019_b39c_3243211a06b7.slice/crio-d33ac8498041af3c60701c9e46d9b45f84db426ac6331119f12403a3c8994891 WatchSource:0}: Error finding container d33ac8498041af3c60701c9e46d9b45f84db426ac6331119f12403a3c8994891: Status 404 returned error can't find the container with id d33ac8498041af3c60701c9e46d9b45f84db426ac6331119f12403a3c8994891 Jul 15 12:45:57.690860 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.690839 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 12:45:57.801847 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.801804 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:57.909128 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:57.909101 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:58.247155 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:58.247126 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" event={"ID":"fcb75fce-3be3-4019-b39c-3243211a06b7","Type":"ContainerStarted","Data":"3e960c2ee52c1e5121c74701cfe0c656c55441dcd820298658539c0ca0f10463"} Jul 15 12:45:58.247504 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:58.247197 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" Jul 15 12:45:58.247504 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:58.247217 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" event={"ID":"fcb75fce-3be3-4019-b39c-3243211a06b7","Type":"ContainerStarted","Data":"d33ac8498041af3c60701c9e46d9b45f84db426ac6331119f12403a3c8994891"} Jul 15 12:45:58.265437 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:45:58.265391 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" podStartSLOduration=0.785857272 podStartE2EDuration="1.265376605s" totalImagesPullingTime="479.519333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:45:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:45:58.262848552 +0000 UTC m=+2081.897243500" watchObservedRunningTime="2026-07-15 12:45:58.265376605 +0000 UTC m=+2081.899771553" Jul 15 12:46:01.771336 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:01.771295 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:02.395089 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:02.395054 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:04.257583 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:04.257558 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" Jul 15 12:46:04.549305 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:04.549226 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:04.603462 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:04.603430 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:08.571793 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:08.571743 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:08.673103 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:08.673060 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:11.236651 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:11.236615 2556 patch_prober.go:28] interesting pod/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 15 12:46:11.237121 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:11.236669 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" podUID="385dd7f4-e237-4b56-bdaa-e25244a624de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 15 12:46:11.308100 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:11.308065 2556 generic.go:356] "Generic (PLEG): container finished" podID="385dd7f4-e237-4b56-bdaa-e25244a624de" containerID="ea104a73a1a7156481da63c2f1461d38e09c4534b6d4963d101db1a75541e249" exitCode=0 Jul 15 12:46:11.308254 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:11.308140 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" event={"ID":"385dd7f4-e237-4b56-bdaa-e25244a624de","Type":"ContainerDied","Data":"ea104a73a1a7156481da63c2f1461d38e09c4534b6d4963d101db1a75541e249"} Jul 15 12:46:11.308528 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:11.308515 2556 scope.go:117] "RemoveContainer" containerID="ea104a73a1a7156481da63c2f1461d38e09c4534b6d4963d101db1a75541e249" Jul 15 12:46:11.309498 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:11.309472 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:46:11.309553 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:11.309543 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:46:11.309605 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:11.309577 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:46:12.317250 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:12.317209 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" event={"ID":"385dd7f4-e237-4b56-bdaa-e25244a624de","Type":"ContainerStarted","Data":"8922b08e29b841d50529768797dc16fdd145cebf3fb1775bbdfc8a48e02dfe7f"} Jul 15 12:46:13.229099 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:13.229062 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:46:13.229313 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:13.229285 2556 patch_prober.go:28] interesting pod/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 15 12:46:13.229367 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:13.229328 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" podUID="385dd7f4-e237-4b56-bdaa-e25244a624de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 15 12:46:14.229004 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:14.228969 2556 patch_prober.go:28] interesting pod/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 15 12:46:14.229387 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:14.229035 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" podUID="385dd7f4-e237-4b56-bdaa-e25244a624de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 15 12:46:15.228990 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:15.228954 2556 patch_prober.go:28] interesting pod/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 15 12:46:15.229182 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:15.229008 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" podUID="385dd7f4-e237-4b56-bdaa-e25244a624de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 15 12:46:16.232503 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:16.232476 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:46:16.232901 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:16.232740 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:46:16.233486 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:16.233469 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:46:16.843079 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:16.843052 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:46:16.848379 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:16.848359 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:46:18.790845 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:18.790809 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:18.890897 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:18.890860 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:19.325954 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:19.325573 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc"] Jul 15 12:46:19.325954 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:19.325829 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" podUID="fcb75fce-3be3-4019-b39c-3243211a06b7" containerName="maas-api" containerID="cri-o://3e960c2ee52c1e5121c74701cfe0c656c55441dcd820298658539c0ca0f10463" gracePeriod=30 Jul 15 12:46:19.346128 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:19.346105 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:19.568950 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:19.568925 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" Jul 15 12:46:19.702208 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:19.702119 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r9xg\" (UniqueName: \"kubernetes.io/projected/fcb75fce-3be3-4019-b39c-3243211a06b7-kube-api-access-6r9xg\") pod \"fcb75fce-3be3-4019-b39c-3243211a06b7\" (UID: \"fcb75fce-3be3-4019-b39c-3243211a06b7\") " Jul 15 12:46:19.702208 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:19.702174 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb75fce-3be3-4019-b39c-3243211a06b7-openshift-service-ca\") pod \"fcb75fce-3be3-4019-b39c-3243211a06b7\" (UID: \"fcb75fce-3be3-4019-b39c-3243211a06b7\") " Jul 15 12:46:19.702208 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:19.702207 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb75fce-3be3-4019-b39c-3243211a06b7-maas-api-tls\") pod \"fcb75fce-3be3-4019-b39c-3243211a06b7\" (UID: \"fcb75fce-3be3-4019-b39c-3243211a06b7\") " Jul 15 12:46:19.702590 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:19.702563 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb75fce-3be3-4019-b39c-3243211a06b7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fcb75fce-3be3-4019-b39c-3243211a06b7" (UID: "fcb75fce-3be3-4019-b39c-3243211a06b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:46:19.704389 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:19.704355 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb75fce-3be3-4019-b39c-3243211a06b7-kube-api-access-6r9xg" (OuterVolumeSpecName: "kube-api-access-6r9xg") pod "fcb75fce-3be3-4019-b39c-3243211a06b7" (UID: "fcb75fce-3be3-4019-b39c-3243211a06b7"). InnerVolumeSpecName "kube-api-access-6r9xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:46:19.704522 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:19.704433 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcb75fce-3be3-4019-b39c-3243211a06b7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fcb75fce-3be3-4019-b39c-3243211a06b7" (UID: "fcb75fce-3be3-4019-b39c-3243211a06b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:46:19.802978 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:19.802944 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r9xg\" (UniqueName: \"kubernetes.io/projected/fcb75fce-3be3-4019-b39c-3243211a06b7-kube-api-access-6r9xg\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:46:19.802978 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:19.802968 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb75fce-3be3-4019-b39c-3243211a06b7-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:46:19.802978 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:19.802979 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb75fce-3be3-4019-b39c-3243211a06b7-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:46:20.353716 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:20.353685 2556 generic.go:356] "Generic (PLEG): container finished" podID="fcb75fce-3be3-4019-b39c-3243211a06b7" containerID="3e960c2ee52c1e5121c74701cfe0c656c55441dcd820298658539c0ca0f10463" exitCode=0 Jul 15 12:46:20.353894 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:20.353733 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" Jul 15 12:46:20.353894 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:20.353790 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" event={"ID":"fcb75fce-3be3-4019-b39c-3243211a06b7","Type":"ContainerDied","Data":"3e960c2ee52c1e5121c74701cfe0c656c55441dcd820298658539c0ca0f10463"} Jul 15 12:46:20.353894 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:20.353839 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc" event={"ID":"fcb75fce-3be3-4019-b39c-3243211a06b7","Type":"ContainerDied","Data":"d33ac8498041af3c60701c9e46d9b45f84db426ac6331119f12403a3c8994891"} Jul 15 12:46:20.353894 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:20.353861 2556 scope.go:117] "RemoveContainer" containerID="3e960c2ee52c1e5121c74701cfe0c656c55441dcd820298658539c0ca0f10463" Jul 15 12:46:20.362680 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:20.362660 2556 scope.go:117] "RemoveContainer" containerID="3e960c2ee52c1e5121c74701cfe0c656c55441dcd820298658539c0ca0f10463" Jul 15 12:46:20.362989 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:46:20.362971 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e960c2ee52c1e5121c74701cfe0c656c55441dcd820298658539c0ca0f10463\": container with ID starting with 3e960c2ee52c1e5121c74701cfe0c656c55441dcd820298658539c0ca0f10463 not found: ID does not exist" containerID="3e960c2ee52c1e5121c74701cfe0c656c55441dcd820298658539c0ca0f10463" Jul 15 12:46:20.363058 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:20.362998 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e960c2ee52c1e5121c74701cfe0c656c55441dcd820298658539c0ca0f10463"} err="failed to get container status \"3e960c2ee52c1e5121c74701cfe0c656c55441dcd820298658539c0ca0f10463\": rpc error: code = NotFound desc = could not find container \"3e960c2ee52c1e5121c74701cfe0c656c55441dcd820298658539c0ca0f10463\": container with ID starting with 3e960c2ee52c1e5121c74701cfe0c656c55441dcd820298658539c0ca0f10463 not found: ID does not exist" Jul 15 12:46:20.374205 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:20.374160 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc"] Jul 15 12:46:20.376132 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:20.376109 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7c0f6bf-6f55bdb6b-2z5zc"] Jul 15 12:46:20.854207 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:20.854174 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcb75fce-3be3-4019-b39c-3243211a06b7" path="/var/lib/kubelet/pods/fcb75fce-3be3-4019-b39c-3243211a06b7/volumes" Jul 15 12:46:24.033886 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:24.033853 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:29.872195 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:29.872157 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:29.972113 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:29.972073 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:34.697388 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:34.697354 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc"] Jul 15 12:46:34.697922 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:34.697613 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" podUID="385dd7f4-e237-4b56-bdaa-e25244a624de" containerName="istio-proxy" containerID="cri-o://8922b08e29b841d50529768797dc16fdd145cebf3fb1775bbdfc8a48e02dfe7f" gracePeriod=30 Jul 15 12:46:34.713094 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:34.713064 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:34.936563 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:34.936532 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:35.065022 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:35.064983 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:35.164145 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:35.164109 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:35.291733 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:35.291692 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:39.952679 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:39.952655 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:46:40.065136 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.065103 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-workload-certs\") pod \"385dd7f4-e237-4b56-bdaa-e25244a624de\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " Jul 15 12:46:40.065321 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.065154 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hnkb\" (UniqueName: \"kubernetes.io/projected/385dd7f4-e237-4b56-bdaa-e25244a624de-kube-api-access-9hnkb\") pod \"385dd7f4-e237-4b56-bdaa-e25244a624de\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " Jul 15 12:46:40.065321 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.065179 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-token\") pod \"385dd7f4-e237-4b56-bdaa-e25244a624de\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " Jul 15 12:46:40.065321 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.065199 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385dd7f4-e237-4b56-bdaa-e25244a624de-istiod-ca-cert\") pod \"385dd7f4-e237-4b56-bdaa-e25244a624de\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " Jul 15 12:46:40.065499 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.065341 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-podinfo\") pod \"385dd7f4-e237-4b56-bdaa-e25244a624de\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " Jul 15 12:46:40.065499 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.065415 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-credential-socket\") pod \"385dd7f4-e237-4b56-bdaa-e25244a624de\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " Jul 15 12:46:40.065499 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.065441 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-workload-socket\") pod \"385dd7f4-e237-4b56-bdaa-e25244a624de\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " Jul 15 12:46:40.065499 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.065473 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-envoy\") pod \"385dd7f4-e237-4b56-bdaa-e25244a624de\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " Jul 15 12:46:40.065711 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.065517 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-data\") pod \"385dd7f4-e237-4b56-bdaa-e25244a624de\" (UID: \"385dd7f4-e237-4b56-bdaa-e25244a624de\") " Jul 15 12:46:40.065711 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.065439 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "385dd7f4-e237-4b56-bdaa-e25244a624de" (UID: "385dd7f4-e237-4b56-bdaa-e25244a624de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:46:40.065711 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.065541 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/385dd7f4-e237-4b56-bdaa-e25244a624de-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "385dd7f4-e237-4b56-bdaa-e25244a624de" (UID: "385dd7f4-e237-4b56-bdaa-e25244a624de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:46:40.065711 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.065633 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "385dd7f4-e237-4b56-bdaa-e25244a624de" (UID: "385dd7f4-e237-4b56-bdaa-e25244a624de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:46:40.066122 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.065831 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-credential-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:46:40.066122 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.065849 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-workload-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:46:40.066122 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.065862 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385dd7f4-e237-4b56-bdaa-e25244a624de-istiod-ca-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:46:40.066288 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.066173 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "385dd7f4-e237-4b56-bdaa-e25244a624de" (UID: "385dd7f4-e237-4b56-bdaa-e25244a624de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:46:40.066590 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.066562 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-data" (OuterVolumeSpecName: "istio-data") pod "385dd7f4-e237-4b56-bdaa-e25244a624de" (UID: "385dd7f4-e237-4b56-bdaa-e25244a624de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:46:40.067508 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.067480 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385dd7f4-e237-4b56-bdaa-e25244a624de-kube-api-access-9hnkb" (OuterVolumeSpecName: "kube-api-access-9hnkb") pod "385dd7f4-e237-4b56-bdaa-e25244a624de" (UID: "385dd7f4-e237-4b56-bdaa-e25244a624de"). InnerVolumeSpecName "kube-api-access-9hnkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:46:40.067508 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.067490 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-token" (OuterVolumeSpecName: "istio-token") pod "385dd7f4-e237-4b56-bdaa-e25244a624de" (UID: "385dd7f4-e237-4b56-bdaa-e25244a624de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:46:40.067678 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.067562 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "385dd7f4-e237-4b56-bdaa-e25244a624de" (UID: "385dd7f4-e237-4b56-bdaa-e25244a624de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:46:40.067847 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.067824 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "385dd7f4-e237-4b56-bdaa-e25244a624de" (UID: "385dd7f4-e237-4b56-bdaa-e25244a624de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:46:40.166410 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.166373 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-workload-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:46:40.166410 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.166405 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-envoy\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:46:40.166410 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.166417 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-data\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:46:40.166656 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.166428 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hnkb\" (UniqueName: \"kubernetes.io/projected/385dd7f4-e237-4b56-bdaa-e25244a624de-kube-api-access-9hnkb\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:46:40.166656 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.166441 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:46:40.166656 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.166453 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385dd7f4-e237-4b56-bdaa-e25244a624de-istio-podinfo\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:46:40.448237 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.448143 2556 generic.go:356] "Generic (PLEG): container finished" podID="385dd7f4-e237-4b56-bdaa-e25244a624de" containerID="8922b08e29b841d50529768797dc16fdd145cebf3fb1775bbdfc8a48e02dfe7f" exitCode=0 Jul 15 12:46:40.448237 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.448193 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" Jul 15 12:46:40.448448 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.448229 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" event={"ID":"385dd7f4-e237-4b56-bdaa-e25244a624de","Type":"ContainerDied","Data":"8922b08e29b841d50529768797dc16fdd145cebf3fb1775bbdfc8a48e02dfe7f"} Jul 15 12:46:40.448448 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.448292 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc" event={"ID":"385dd7f4-e237-4b56-bdaa-e25244a624de","Type":"ContainerDied","Data":"c68e6a4e91791414982075c9a09cbfcc9700422450ecd6b3cd6fe157dd1506b9"} Jul 15 12:46:40.448448 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.448316 2556 scope.go:117] "RemoveContainer" containerID="8922b08e29b841d50529768797dc16fdd145cebf3fb1775bbdfc8a48e02dfe7f" Jul 15 12:46:40.457575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.457558 2556 scope.go:117] "RemoveContainer" containerID="ea104a73a1a7156481da63c2f1461d38e09c4534b6d4963d101db1a75541e249" Jul 15 12:46:40.464993 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.464977 2556 scope.go:117] "RemoveContainer" containerID="8922b08e29b841d50529768797dc16fdd145cebf3fb1775bbdfc8a48e02dfe7f" Jul 15 12:46:40.465234 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:46:40.465218 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8922b08e29b841d50529768797dc16fdd145cebf3fb1775bbdfc8a48e02dfe7f\": container with ID starting with 8922b08e29b841d50529768797dc16fdd145cebf3fb1775bbdfc8a48e02dfe7f not found: ID does not exist" containerID="8922b08e29b841d50529768797dc16fdd145cebf3fb1775bbdfc8a48e02dfe7f" Jul 15 12:46:40.465278 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.465241 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8922b08e29b841d50529768797dc16fdd145cebf3fb1775bbdfc8a48e02dfe7f"} err="failed to get container status \"8922b08e29b841d50529768797dc16fdd145cebf3fb1775bbdfc8a48e02dfe7f\": rpc error: code = NotFound desc = could not find container \"8922b08e29b841d50529768797dc16fdd145cebf3fb1775bbdfc8a48e02dfe7f\": container with ID starting with 8922b08e29b841d50529768797dc16fdd145cebf3fb1775bbdfc8a48e02dfe7f not found: ID does not exist" Jul 15 12:46:40.465278 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.465256 2556 scope.go:117] "RemoveContainer" containerID="ea104a73a1a7156481da63c2f1461d38e09c4534b6d4963d101db1a75541e249" Jul 15 12:46:40.465492 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:46:40.465473 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea104a73a1a7156481da63c2f1461d38e09c4534b6d4963d101db1a75541e249\": container with ID starting with ea104a73a1a7156481da63c2f1461d38e09c4534b6d4963d101db1a75541e249 not found: ID does not exist" containerID="ea104a73a1a7156481da63c2f1461d38e09c4534b6d4963d101db1a75541e249" Jul 15 12:46:40.465550 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.465501 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea104a73a1a7156481da63c2f1461d38e09c4534b6d4963d101db1a75541e249"} err="failed to get container status \"ea104a73a1a7156481da63c2f1461d38e09c4534b6d4963d101db1a75541e249\": rpc error: code = NotFound desc = could not find container \"ea104a73a1a7156481da63c2f1461d38e09c4534b6d4963d101db1a75541e249\": container with ID starting with ea104a73a1a7156481da63c2f1461d38e09c4534b6d4963d101db1a75541e249 not found: ID does not exist" Jul 15 12:46:40.469096 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.469077 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc"] Jul 15 12:46:40.471264 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.471245 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c7c0f6bf-openshift-default-7c5479d687-59nhc"] Jul 15 12:46:40.854530 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.854490 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="385dd7f4-e237-4b56-bdaa-e25244a624de" path="/var/lib/kubelet/pods/385dd7f4-e237-4b56-bdaa-e25244a624de/volumes" Jul 15 12:46:40.959799 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:40.959743 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:41.065494 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:41.065461 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:41.164197 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:41.164110 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:41.315662 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:41.315628 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:47.027905 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:47.027873 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:47.110256 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:47.110227 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:53.839285 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:53.839250 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:54.133040 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.132956 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr"] Jul 15 12:46:54.133377 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.133356 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcb75fce-3be3-4019-b39c-3243211a06b7" containerName="maas-api" Jul 15 12:46:54.133487 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.133380 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcb75fce-3be3-4019-b39c-3243211a06b7" containerName="maas-api" Jul 15 12:46:54.133487 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.133411 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="385dd7f4-e237-4b56-bdaa-e25244a624de" containerName="istio-proxy" Jul 15 12:46:54.133487 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.133420 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="385dd7f4-e237-4b56-bdaa-e25244a624de" containerName="istio-proxy" Jul 15 12:46:54.133487 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.133432 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="385dd7f4-e237-4b56-bdaa-e25244a624de" containerName="istio-proxy" Jul 15 12:46:54.133487 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.133440 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="385dd7f4-e237-4b56-bdaa-e25244a624de" containerName="istio-proxy" Jul 15 12:46:54.133738 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.133539 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcb75fce-3be3-4019-b39c-3243211a06b7" containerName="maas-api" Jul 15 12:46:54.133738 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.133551 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="385dd7f4-e237-4b56-bdaa-e25244a624de" containerName="istio-proxy" Jul 15 12:46:54.133738 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.133562 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="385dd7f4-e237-4b56-bdaa-e25244a624de" containerName="istio-proxy" Jul 15 12:46:54.136540 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.136518 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.139330 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.139306 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e1773daa-openshift-default-dockercfg-4rrvp\"" Jul 15 12:46:54.144995 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.144972 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr"] Jul 15 12:46:54.159268 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.159234 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:54.279738 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.279703 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-data\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.279738 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.279736 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-envoy\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.279963 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.279761 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-credential-socket\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.279963 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.279827 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-workload-certs\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.279963 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.279853 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnfgf\" (UniqueName: \"kubernetes.io/projected/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-kube-api-access-mnfgf\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.279963 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.279915 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-podinfo\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.279963 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.279934 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-workload-socket\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.279963 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.279948 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-token\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.280175 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.279994 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istiod-ca-cert\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.381142 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.381106 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-workload-certs\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.381142 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.381141 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnfgf\" (UniqueName: \"kubernetes.io/projected/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-kube-api-access-mnfgf\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.381377 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.381190 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-podinfo\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.381377 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.381208 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-workload-socket\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.381377 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.381222 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-token\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.381377 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.381261 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istiod-ca-cert\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.381377 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.381286 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-data\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.381377 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.381312 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-envoy\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.381377 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.381360 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-credential-socket\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.381736 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.381567 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-workload-certs\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.381736 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.381585 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-workload-socket\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.381905 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.381817 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-credential-socket\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.382000 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.381981 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-data\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.382283 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.382261 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istiod-ca-cert\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.383739 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.383688 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-envoy\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.383871 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.383854 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-podinfo\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.391333 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.391303 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-token\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.391917 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.391894 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnfgf\" (UniqueName: \"kubernetes.io/projected/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-kube-api-access-mnfgf\") pod \"e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.435683 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.435652 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:54.448703 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.448684 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:54.590055 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.589907 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr"] Jul 15 12:46:54.592752 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:46:54.592720 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1571ce9_bff3_4ce9_9872_bd01e39d06cd.slice/crio-268306d5210f2835a8e5646bbccb8797614aa8e49d11d1e224498331cc84072e WatchSource:0}: Error finding container 268306d5210f2835a8e5646bbccb8797614aa8e49d11d1e224498331cc84072e: Status 404 returned error can't find the container with id 268306d5210f2835a8e5646bbccb8797614aa8e49d11d1e224498331cc84072e Jul 15 12:46:54.595179 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.595137 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:46:54.595274 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.595225 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:46:54.595274 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:54.595266 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:46:55.526357 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:55.526325 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" event={"ID":"c1571ce9-bff3-4ce9-9872-bd01e39d06cd","Type":"ContainerStarted","Data":"ad81be044dc9343c01bcaff9f146c597fec5784a31a272d0847d5a415b5949b6"} Jul 15 12:46:55.526748 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:55.526399 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" event={"ID":"c1571ce9-bff3-4ce9-9872-bd01e39d06cd","Type":"ContainerStarted","Data":"268306d5210f2835a8e5646bbccb8797614aa8e49d11d1e224498331cc84072e"} Jul 15 12:46:55.551836 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:55.551790 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" podStartSLOduration=1.551761583 podStartE2EDuration="1.551761583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:46:55.549538635 +0000 UTC m=+2139.183933583" watchObservedRunningTime="2026-07-15 12:46:55.551761583 +0000 UTC m=+2139.186156530" Jul 15 12:46:56.449262 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:56.449227 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:56.454057 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:56.454031 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:56.534531 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:56.534500 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:56.535396 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:56.535380 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:46:57.472460 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:57.472419 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:57.570583 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:57.570550 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:57.723427 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:57.723343 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:57.982790 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:57.982693 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:58.274358 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:58.274321 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:58.376348 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:46:58.376316 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:02.000856 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:02.000825 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:11.819418 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:11.819381 2556 patch_prober.go:28] interesting pod/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": read tcp 10.134.0.2:47408->10.134.0.37:15021: read: connection reset by peer" start-of-body= Jul 15 12:47:11.819847 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:11.819466 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" podUID="c1571ce9-bff3-4ce9-9872-bd01e39d06cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": read tcp 10.134.0.2:47408->10.134.0.37:15021: read: connection reset by peer" Jul 15 12:47:12.607631 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:12.607597 2556 generic.go:356] "Generic (PLEG): container finished" podID="c1571ce9-bff3-4ce9-9872-bd01e39d06cd" containerID="ad81be044dc9343c01bcaff9f146c597fec5784a31a272d0847d5a415b5949b6" exitCode=0 Jul 15 12:47:12.607815 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:12.607661 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" event={"ID":"c1571ce9-bff3-4ce9-9872-bd01e39d06cd","Type":"ContainerDied","Data":"ad81be044dc9343c01bcaff9f146c597fec5784a31a272d0847d5a415b5949b6"} Jul 15 12:47:12.608052 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:12.608035 2556 scope.go:117] "RemoveContainer" containerID="ad81be044dc9343c01bcaff9f146c597fec5784a31a272d0847d5a415b5949b6" Jul 15 12:47:12.609009 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:12.608975 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:47:12.609119 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:12.609062 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:47:12.609184 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:12.609121 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:47:13.577854 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:13.577812 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:13.616575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:13.616530 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" event={"ID":"c1571ce9-bff3-4ce9-9872-bd01e39d06cd","Type":"ContainerStarted","Data":"c1e3d72acff9c50f04612b9d9dc57d0444e567431a9b437ec1e2f7ffdd3df9cb"} Jul 15 12:47:14.449324 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:14.449289 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:47:14.449518 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:14.449483 2556 patch_prober.go:28] interesting pod/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 15 12:47:14.449570 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:14.449542 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" podUID="c1571ce9-bff3-4ce9-9872-bd01e39d06cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 15 12:47:15.449200 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:15.449164 2556 patch_prober.go:28] interesting pod/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 15 12:47:15.449588 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:15.449217 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" podUID="c1571ce9-bff3-4ce9-9872-bd01e39d06cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 15 12:47:16.449683 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:16.449650 2556 patch_prober.go:28] interesting pod/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 15 12:47:16.450094 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:16.449703 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" podUID="c1571ce9-bff3-4ce9-9872-bd01e39d06cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 15 12:47:16.921667 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:16.921625 2556 scope.go:117] "RemoveContainer" containerID="63d97e3bd599c54e4171768f17aa8325e4b6e56fd8043e857c30c1fff4c0b490" Jul 15 12:47:17.449127 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:17.449094 2556 patch_prober.go:28] interesting pod/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 15 12:47:17.449315 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:17.449159 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" podUID="c1571ce9-bff3-4ce9-9872-bd01e39d06cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 15 12:47:18.453279 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:18.453249 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:47:18.453750 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:18.453576 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:47:18.454332 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:18.454316 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:47:50.687369 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:50.687336 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:55.409445 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:47:55.409415 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:00.388500 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:48:00.388471 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:16.949006 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:48:16.948975 2556 scope.go:117] "RemoveContainer" containerID="aae5bdb4a09a73245dfc8450eece27c69999ebce2b1887e857abb3b39850ea21" Jul 15 12:48:19.137037 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:48:19.137001 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:19.238365 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:48:19.238327 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:37.278427 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:48:37.278384 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:37.405021 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:48:37.404992 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:42.861281 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:48:42.861250 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:48.447269 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:48:48.447236 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:48.507882 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:48:48.507842 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:57.601252 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:48:57.601217 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr"] Jul 15 12:48:57.601839 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:48:57.601471 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" podUID="c1571ce9-bff3-4ce9-9872-bd01e39d06cd" containerName="istio-proxy" containerID="cri-o://c1e3d72acff9c50f04612b9d9dc57d0444e567431a9b437ec1e2f7ffdd3df9cb" gracePeriod=30 Jul 15 12:48:57.625556 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:48:57.625530 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:57.849423 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:48:57.849381 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:58.910302 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:48:58.910267 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:59.002632 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:48:59.002593 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:59.101656 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:48:59.101627 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:59.825502 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:48:59.825471 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:59.925483 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:48:59.925451 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:00.023105 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:00.023073 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:01.419572 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:01.419541 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:02.847928 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.847904 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:49:02.955822 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.955738 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-data\") pod \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " Jul 15 12:49:02.955822 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.955816 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-token\") pod \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " Jul 15 12:49:02.956138 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.955845 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-workload-certs\") pod \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " Jul 15 12:49:02.956138 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.955864 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-credential-socket\") pod \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " Jul 15 12:49:02.956138 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.955911 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-podinfo\") pod \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " Jul 15 12:49:02.956138 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.955941 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnfgf\" (UniqueName: \"kubernetes.io/projected/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-kube-api-access-mnfgf\") pod \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " Jul 15 12:49:02.956138 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.955983 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-workload-socket\") pod \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " Jul 15 12:49:02.956138 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.956021 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-envoy\") pod \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " Jul 15 12:49:02.956138 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.956044 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istiod-ca-cert\") pod \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\" (UID: \"c1571ce9-bff3-4ce9-9872-bd01e39d06cd\") " Jul 15 12:49:02.956138 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.956102 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c1571ce9-bff3-4ce9-9872-bd01e39d06cd" (UID: "c1571ce9-bff3-4ce9-9872-bd01e39d06cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:49:02.956567 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.956207 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c1571ce9-bff3-4ce9-9872-bd01e39d06cd" (UID: "c1571ce9-bff3-4ce9-9872-bd01e39d06cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:49:02.956567 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.956316 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c1571ce9-bff3-4ce9-9872-bd01e39d06cd" (UID: "c1571ce9-bff3-4ce9-9872-bd01e39d06cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:49:02.956567 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.956383 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-workload-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:49:02.956567 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.956402 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-workload-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:49:02.956567 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.956418 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-credential-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:49:02.956567 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.956500 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c1571ce9-bff3-4ce9-9872-bd01e39d06cd" (UID: "c1571ce9-bff3-4ce9-9872-bd01e39d06cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:49:02.957069 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.956947 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-data" (OuterVolumeSpecName: "istio-data") pod "c1571ce9-bff3-4ce9-9872-bd01e39d06cd" (UID: "c1571ce9-bff3-4ce9-9872-bd01e39d06cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:49:02.958411 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.958382 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c1571ce9-bff3-4ce9-9872-bd01e39d06cd" (UID: "c1571ce9-bff3-4ce9-9872-bd01e39d06cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:49:02.958411 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.958398 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-kube-api-access-mnfgf" (OuterVolumeSpecName: "kube-api-access-mnfgf") pod "c1571ce9-bff3-4ce9-9872-bd01e39d06cd" (UID: "c1571ce9-bff3-4ce9-9872-bd01e39d06cd"). InnerVolumeSpecName "kube-api-access-mnfgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:49:02.958573 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.958422 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c1571ce9-bff3-4ce9-9872-bd01e39d06cd" (UID: "c1571ce9-bff3-4ce9-9872-bd01e39d06cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:49:02.958680 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:02.958660 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-token" (OuterVolumeSpecName: "istio-token") pod "c1571ce9-bff3-4ce9-9872-bd01e39d06cd" (UID: "c1571ce9-bff3-4ce9-9872-bd01e39d06cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:49:03.057191 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:03.057157 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-envoy\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:49:03.057191 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:03.057185 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istiod-ca-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:49:03.057191 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:03.057197 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-data\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:49:03.057396 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:03.057208 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:49:03.057396 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:03.057220 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-istio-podinfo\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:49:03.057396 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:03.057230 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnfgf\" (UniqueName: \"kubernetes.io/projected/c1571ce9-bff3-4ce9-9872-bd01e39d06cd-kube-api-access-mnfgf\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:49:03.101608 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:03.101576 2556 generic.go:356] "Generic (PLEG): container finished" podID="c1571ce9-bff3-4ce9-9872-bd01e39d06cd" containerID="c1e3d72acff9c50f04612b9d9dc57d0444e567431a9b437ec1e2f7ffdd3df9cb" exitCode=0 Jul 15 12:49:03.101741 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:03.101628 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" Jul 15 12:49:03.101741 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:03.101655 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" event={"ID":"c1571ce9-bff3-4ce9-9872-bd01e39d06cd","Type":"ContainerDied","Data":"c1e3d72acff9c50f04612b9d9dc57d0444e567431a9b437ec1e2f7ffdd3df9cb"} Jul 15 12:49:03.101741 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:03.101709 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr" event={"ID":"c1571ce9-bff3-4ce9-9872-bd01e39d06cd","Type":"ContainerDied","Data":"268306d5210f2835a8e5646bbccb8797614aa8e49d11d1e224498331cc84072e"} Jul 15 12:49:03.101868 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:03.101731 2556 scope.go:117] "RemoveContainer" containerID="c1e3d72acff9c50f04612b9d9dc57d0444e567431a9b437ec1e2f7ffdd3df9cb" Jul 15 12:49:03.111508 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:03.111493 2556 scope.go:117] "RemoveContainer" containerID="ad81be044dc9343c01bcaff9f146c597fec5784a31a272d0847d5a415b5949b6" Jul 15 12:49:03.119101 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:03.119084 2556 scope.go:117] "RemoveContainer" containerID="c1e3d72acff9c50f04612b9d9dc57d0444e567431a9b437ec1e2f7ffdd3df9cb" Jul 15 12:49:03.119340 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:49:03.119321 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1e3d72acff9c50f04612b9d9dc57d0444e567431a9b437ec1e2f7ffdd3df9cb\": container with ID starting with c1e3d72acff9c50f04612b9d9dc57d0444e567431a9b437ec1e2f7ffdd3df9cb not found: ID does not exist" containerID="c1e3d72acff9c50f04612b9d9dc57d0444e567431a9b437ec1e2f7ffdd3df9cb" Jul 15 12:49:03.119397 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:03.119349 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e3d72acff9c50f04612b9d9dc57d0444e567431a9b437ec1e2f7ffdd3df9cb"} err="failed to get container status \"c1e3d72acff9c50f04612b9d9dc57d0444e567431a9b437ec1e2f7ffdd3df9cb\": rpc error: code = NotFound desc = could not find container \"c1e3d72acff9c50f04612b9d9dc57d0444e567431a9b437ec1e2f7ffdd3df9cb\": container with ID starting with c1e3d72acff9c50f04612b9d9dc57d0444e567431a9b437ec1e2f7ffdd3df9cb not found: ID does not exist" Jul 15 12:49:03.119397 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:03.119367 2556 scope.go:117] "RemoveContainer" containerID="ad81be044dc9343c01bcaff9f146c597fec5784a31a272d0847d5a415b5949b6" Jul 15 12:49:03.119591 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:49:03.119576 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad81be044dc9343c01bcaff9f146c597fec5784a31a272d0847d5a415b5949b6\": container with ID starting with ad81be044dc9343c01bcaff9f146c597fec5784a31a272d0847d5a415b5949b6 not found: ID does not exist" containerID="ad81be044dc9343c01bcaff9f146c597fec5784a31a272d0847d5a415b5949b6" Jul 15 12:49:03.119632 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:03.119594 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad81be044dc9343c01bcaff9f146c597fec5784a31a272d0847d5a415b5949b6"} err="failed to get container status \"ad81be044dc9343c01bcaff9f146c597fec5784a31a272d0847d5a415b5949b6\": rpc error: code = NotFound desc = could not find container \"ad81be044dc9343c01bcaff9f146c597fec5784a31a272d0847d5a415b5949b6\": container with ID starting with ad81be044dc9343c01bcaff9f146c597fec5784a31a272d0847d5a415b5949b6 not found: ID does not exist" Jul 15 12:49:03.127207 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:03.127186 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr"] Jul 15 12:49:03.130678 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:03.130659 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e1773daa-openshift-default-684b4db5c5-jjgzr"] Jul 15 12:49:04.854182 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:04.854150 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1571ce9-bff3-4ce9-9872-bd01e39d06cd" path="/var/lib/kubelet/pods/c1571ce9-bff3-4ce9-9872-bd01e39d06cd/volumes" Jul 15 12:49:07.138059 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:07.138027 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:07.193347 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:07.193317 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:07.815202 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:07.815167 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:13.981292 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:13.981259 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:14.033001 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:14.032974 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:24.502281 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:24.502228 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:24.859225 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:24.859134 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:45.847848 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:45.847810 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:48.729358 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:48.729318 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:51.453923 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:51.453886 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:51.555309 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:51.555269 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:51.807753 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:51.807715 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:52.126586 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.126512 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t"] Jul 15 12:49:52.126828 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.126815 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1571ce9-bff3-4ce9-9872-bd01e39d06cd" containerName="istio-proxy" Jul 15 12:49:52.126894 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.126831 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1571ce9-bff3-4ce9-9872-bd01e39d06cd" containerName="istio-proxy" Jul 15 12:49:52.126894 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.126842 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1571ce9-bff3-4ce9-9872-bd01e39d06cd" containerName="istio-proxy" Jul 15 12:49:52.126894 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.126847 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1571ce9-bff3-4ce9-9872-bd01e39d06cd" containerName="istio-proxy" Jul 15 12:49:52.127000 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.126904 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1571ce9-bff3-4ce9-9872-bd01e39d06cd" containerName="istio-proxy" Jul 15 12:49:52.127000 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.126912 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1571ce9-bff3-4ce9-9872-bd01e39d06cd" containerName="istio-proxy" Jul 15 12:49:52.129610 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.129594 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.132351 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.132331 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-99c883-openshift-default-dockercfg-gvkc9\"" Jul 15 12:49:52.141954 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.141933 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t"] Jul 15 12:49:52.150580 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.150560 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:52.226374 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.226336 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-envoy\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.226550 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.226380 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istiod-ca-cert\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.226550 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.226403 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-credential-socket\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.226550 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.226424 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-token\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.226550 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.226443 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f5bl\" (UniqueName: \"kubernetes.io/projected/cdbc4a2b-d869-48fe-834e-2233cfbe840d-kube-api-access-8f5bl\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.226550 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.226460 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-workload-socket\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.226550 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.226500 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-data\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.226550 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.226527 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-podinfo\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.226962 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.226585 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-workload-certs\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.327852 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.327819 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-credential-socket\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.327852 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.327854 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-token\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.327852 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.327873 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f5bl\" (UniqueName: \"kubernetes.io/projected/cdbc4a2b-d869-48fe-834e-2233cfbe840d-kube-api-access-8f5bl\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.328201 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.327894 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-workload-socket\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.328201 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.327929 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-data\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.328201 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.327953 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-podinfo\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.328201 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.328018 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-workload-certs\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.328201 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.328062 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-envoy\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.328201 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.328095 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istiod-ca-cert\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.328602 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.328335 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-credential-socket\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.328602 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.328362 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-data\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.328958 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.328932 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-workload-socket\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.329256 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.329104 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-workload-certs\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.329327 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.329263 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istiod-ca-cert\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.330735 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.330714 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-envoy\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.331176 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.331153 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-podinfo\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.338735 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.338716 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-token\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.339236 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.339218 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f5bl\" (UniqueName: \"kubernetes.io/projected/cdbc4a2b-d869-48fe-834e-2233cfbe840d-kube-api-access-8f5bl\") pod \"e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.371726 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.371690 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:52.439083 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.439000 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:52.582030 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.582005 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t"] Jul 15 12:49:52.584528 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:49:52.584502 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdbc4a2b_d869_48fe_834e_2233cfbe840d.slice/crio-eeeb7debdc4d070bcf8a83d98bcbb0dfbbc1e4c39acc38e5a402c23a643057f8 WatchSource:0}: Error finding container eeeb7debdc4d070bcf8a83d98bcbb0dfbbc1e4c39acc38e5a402c23a643057f8: Status 404 returned error can't find the container with id eeeb7debdc4d070bcf8a83d98bcbb0dfbbc1e4c39acc38e5a402c23a643057f8 Jul 15 12:49:52.586530 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.586498 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:49:52.586589 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.586567 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:49:52.586624 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:52.586617 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:49:53.320958 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:53.320926 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" event={"ID":"cdbc4a2b-d869-48fe-834e-2233cfbe840d","Type":"ContainerStarted","Data":"cfb063102a6f7310ea22ce62fb94555090dc9d3357142fe7744569ad3d36c8c9"} Jul 15 12:49:53.321151 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:53.320976 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" event={"ID":"cdbc4a2b-d869-48fe-834e-2233cfbe840d","Type":"ContainerStarted","Data":"eeeb7debdc4d070bcf8a83d98bcbb0dfbbc1e4c39acc38e5a402c23a643057f8"} Jul 15 12:49:53.341842 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:53.341795 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" podStartSLOduration=1.341762462 podStartE2EDuration="1.341762462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:49:53.339439776 +0000 UTC m=+2316.973834726" watchObservedRunningTime="2026-07-15 12:49:53.341762462 +0000 UTC m=+2316.976157410" Jul 15 12:49:53.439651 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:53.439622 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:53.444354 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:53.444326 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:54.325100 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:54.325072 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:54.326106 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:54.326083 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:49:55.248434 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.248403 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:55.327907 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.327880 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq"] Jul 15 12:49:55.331537 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.331514 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" Jul 15 12:49:55.332204 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.332178 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:55.334127 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.334107 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-99c883-serving-cert\"" Jul 15 12:49:55.334231 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.334141 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 12:49:55.334231 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.334171 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bnnf9\"" Jul 15 12:49:55.335203 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.335188 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 12:49:55.342346 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.342329 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq"] Jul 15 12:49:55.410333 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.410299 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:55.453059 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.453035 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18185de2-f20c-4783-b04a-e32a96d74117-maas-api-tls\") pod \"maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq\" (UID: \"18185de2-f20c-4783-b04a-e32a96d74117\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" Jul 15 12:49:55.453224 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.453085 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18185de2-f20c-4783-b04a-e32a96d74117-openshift-service-ca\") pod \"maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq\" (UID: \"18185de2-f20c-4783-b04a-e32a96d74117\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" Jul 15 12:49:55.453224 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.453104 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt6nw\" (UniqueName: \"kubernetes.io/projected/18185de2-f20c-4783-b04a-e32a96d74117-kube-api-access-bt6nw\") pod \"maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq\" (UID: \"18185de2-f20c-4783-b04a-e32a96d74117\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" Jul 15 12:49:55.551051 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.550960 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:55.554074 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.554053 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18185de2-f20c-4783-b04a-e32a96d74117-openshift-service-ca\") pod \"maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq\" (UID: \"18185de2-f20c-4783-b04a-e32a96d74117\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" Jul 15 12:49:55.554134 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.554085 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6nw\" (UniqueName: \"kubernetes.io/projected/18185de2-f20c-4783-b04a-e32a96d74117-kube-api-access-bt6nw\") pod \"maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq\" (UID: \"18185de2-f20c-4783-b04a-e32a96d74117\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" Jul 15 12:49:55.554171 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.554149 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18185de2-f20c-4783-b04a-e32a96d74117-maas-api-tls\") pod \"maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq\" (UID: \"18185de2-f20c-4783-b04a-e32a96d74117\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" Jul 15 12:49:55.554817 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.554798 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18185de2-f20c-4783-b04a-e32a96d74117-openshift-service-ca\") pod \"maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq\" (UID: \"18185de2-f20c-4783-b04a-e32a96d74117\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" Jul 15 12:49:55.556680 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.556661 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18185de2-f20c-4783-b04a-e32a96d74117-maas-api-tls\") pod \"maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq\" (UID: \"18185de2-f20c-4783-b04a-e32a96d74117\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" Jul 15 12:49:55.562453 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.562427 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt6nw\" (UniqueName: \"kubernetes.io/projected/18185de2-f20c-4783-b04a-e32a96d74117-kube-api-access-bt6nw\") pod \"maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq\" (UID: \"18185de2-f20c-4783-b04a-e32a96d74117\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" Jul 15 12:49:55.643162 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.643125 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" Jul 15 12:49:55.783133 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.783109 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq"] Jul 15 12:49:55.784924 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:49:55.784891 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18185de2_f20c_4783_b04a_e32a96d74117.slice/crio-76f8f5c5c93be15c3b368bdb1f473b587beb4ce6e93470b95cfa4026dc66afa9 WatchSource:0}: Error finding container 76f8f5c5c93be15c3b368bdb1f473b587beb4ce6e93470b95cfa4026dc66afa9: Status 404 returned error can't find the container with id 76f8f5c5c93be15c3b368bdb1f473b587beb4ce6e93470b95cfa4026dc66afa9 Jul 15 12:49:55.811654 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.811572 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:55.913591 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:55.913555 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:56.341721 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:56.341684 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" event={"ID":"18185de2-f20c-4783-b04a-e32a96d74117","Type":"ContainerStarted","Data":"371ba374b311f9096aef2b67570dbec4e70425ed7bc8444f5a4efb4283ead6e3"} Jul 15 12:49:56.342148 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:56.341768 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" event={"ID":"18185de2-f20c-4783-b04a-e32a96d74117","Type":"ContainerStarted","Data":"76f8f5c5c93be15c3b368bdb1f473b587beb4ce6e93470b95cfa4026dc66afa9"} Jul 15 12:49:56.342148 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:56.341951 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" Jul 15 12:49:56.360394 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:56.360346 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" podStartSLOduration=0.870912863 podStartE2EDuration="1.360332607s" totalImagesPullingTime="489.419744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:49:56.358170237 +0000 UTC m=+2319.992565185" watchObservedRunningTime="2026-07-15 12:49:56.360332607 +0000 UTC m=+2319.994727585" Jul 15 12:49:58.214317 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:58.214265 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:59.766497 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:49:59.766446 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:00.116916 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:00.116828 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:00.440131 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:00.440047 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:00.749166 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:00.749137 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:02.351711 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:02.351679 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" Jul 15 12:50:03.281456 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:03.281415 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k"] Jul 15 12:50:03.285395 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:03.285371 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" Jul 15 12:50:03.288176 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:03.288154 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0eb9b9-serving-cert\"" Jul 15 12:50:03.296474 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:03.296450 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k"] Jul 15 12:50:03.318940 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:03.318914 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8sgl\" (UniqueName: \"kubernetes.io/projected/956499ca-1933-4efa-b5e0-0744799baff6-kube-api-access-l8sgl\") pod \"maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k\" (UID: \"956499ca-1933-4efa-b5e0-0744799baff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" Jul 15 12:50:03.319092 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:03.318951 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/956499ca-1933-4efa-b5e0-0744799baff6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k\" (UID: \"956499ca-1933-4efa-b5e0-0744799baff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" Jul 15 12:50:03.319092 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:03.319015 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/956499ca-1933-4efa-b5e0-0744799baff6-maas-api-tls\") pod \"maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k\" (UID: \"956499ca-1933-4efa-b5e0-0744799baff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" Jul 15 12:50:03.420368 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:03.420336 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/956499ca-1933-4efa-b5e0-0744799baff6-maas-api-tls\") pod \"maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k\" (UID: \"956499ca-1933-4efa-b5e0-0744799baff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" Jul 15 12:50:03.420831 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:03.420398 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8sgl\" (UniqueName: \"kubernetes.io/projected/956499ca-1933-4efa-b5e0-0744799baff6-kube-api-access-l8sgl\") pod \"maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k\" (UID: \"956499ca-1933-4efa-b5e0-0744799baff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" Jul 15 12:50:03.420831 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:03.420419 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/956499ca-1933-4efa-b5e0-0744799baff6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k\" (UID: \"956499ca-1933-4efa-b5e0-0744799baff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" Jul 15 12:50:03.420831 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:50:03.420491 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-serving-cert: secret "maas-api-e2e-inf-b-0eb9b9-serving-cert" not found Jul 15 12:50:03.420831 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:50:03.420563 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/956499ca-1933-4efa-b5e0-0744799baff6-maas-api-tls podName:956499ca-1933-4efa-b5e0-0744799baff6 nodeName:}" failed. No retries permitted until 2026-07-15 12:50:03.920544991 +0000 UTC m=+2327.554939919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/956499ca-1933-4efa-b5e0-0744799baff6-maas-api-tls") pod "maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" (UID: "956499ca-1933-4efa-b5e0-0744799baff6") : secret "maas-api-e2e-inf-b-0eb9b9-serving-cert" not found Jul 15 12:50:03.421060 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:03.421042 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/956499ca-1933-4efa-b5e0-0744799baff6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k\" (UID: \"956499ca-1933-4efa-b5e0-0744799baff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" Jul 15 12:50:03.429841 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:03.429808 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8sgl\" (UniqueName: \"kubernetes.io/projected/956499ca-1933-4efa-b5e0-0744799baff6-kube-api-access-l8sgl\") pod \"maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k\" (UID: \"956499ca-1933-4efa-b5e0-0744799baff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" Jul 15 12:50:03.583171 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:03.583091 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:03.684410 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:03.684372 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:03.924375 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:03.924337 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/956499ca-1933-4efa-b5e0-0744799baff6-maas-api-tls\") pod \"maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k\" (UID: \"956499ca-1933-4efa-b5e0-0744799baff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" Jul 15 12:50:03.927220 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:03.927189 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/956499ca-1933-4efa-b5e0-0744799baff6-maas-api-tls\") pod \"maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k\" (UID: \"956499ca-1933-4efa-b5e0-0744799baff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" Jul 15 12:50:04.197234 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:04.197129 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" Jul 15 12:50:04.366584 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:04.366558 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k"] Jul 15 12:50:04.368612 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:50:04.368574 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956499ca_1933_4efa_b5e0_0744799baff6.slice/crio-8904bf07cb5c8989b8c534fe00b8c5f1047c9bd47ac389601cab9e4c2c5f8b21 WatchSource:0}: Error finding container 8904bf07cb5c8989b8c534fe00b8c5f1047c9bd47ac389601cab9e4c2c5f8b21: Status 404 returned error can't find the container with id 8904bf07cb5c8989b8c534fe00b8c5f1047c9bd47ac389601cab9e4c2c5f8b21 Jul 15 12:50:04.380600 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:04.380574 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" event={"ID":"956499ca-1933-4efa-b5e0-0744799baff6","Type":"ContainerStarted","Data":"8904bf07cb5c8989b8c534fe00b8c5f1047c9bd47ac389601cab9e4c2c5f8b21"} Jul 15 12:50:04.419965 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:04.419560 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:04.513559 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:04.513472 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:06.392651 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:06.392607 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" event={"ID":"956499ca-1933-4efa-b5e0-0744799baff6","Type":"ContainerStarted","Data":"e7a2d5e4f84b37a2f3096be03f4aa5424e4fcc4cdc3137e8cec6f3b21a86ce31"} Jul 15 12:50:06.393454 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:06.393427 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" Jul 15 12:50:06.411804 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:06.411381 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" podStartSLOduration=2.407885264 podStartE2EDuration="3.41135828s" totalImagesPullingTime="1.003473016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:50:06.409128387 +0000 UTC m=+2330.043523336" watchObservedRunningTime="2026-07-15 12:50:06.41135828 +0000 UTC m=+2330.045753231" Jul 15 12:50:07.686869 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:07.686757 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:10.367022 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:10.366967 2556 patch_prober.go:28] interesting pod/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 12:50:10.367412 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:10.367050 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" podUID="cdbc4a2b-d869-48fe-834e-2233cfbe840d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 12:50:11.420485 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:11.420449 2556 generic.go:356] "Generic (PLEG): container finished" podID="cdbc4a2b-d869-48fe-834e-2233cfbe840d" containerID="cfb063102a6f7310ea22ce62fb94555090dc9d3357142fe7744569ad3d36c8c9" exitCode=0 Jul 15 12:50:11.420922 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:11.420525 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" event={"ID":"cdbc4a2b-d869-48fe-834e-2233cfbe840d","Type":"ContainerDied","Data":"cfb063102a6f7310ea22ce62fb94555090dc9d3357142fe7744569ad3d36c8c9"} Jul 15 12:50:11.420998 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:11.420979 2556 scope.go:117] "RemoveContainer" containerID="cfb063102a6f7310ea22ce62fb94555090dc9d3357142fe7744569ad3d36c8c9" Jul 15 12:50:11.421992 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:11.421959 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:50:11.422070 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:11.422036 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:50:11.422115 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:11.422073 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:50:12.427977 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:12.427939 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" event={"ID":"cdbc4a2b-d869-48fe-834e-2233cfbe840d","Type":"ContainerStarted","Data":"4a3b76a0f2f6e1b335fb3b7f23df7656435146294ffa66b0bd241ad5953e83a0"} Jul 15 12:50:12.439638 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:12.439620 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:50:12.439878 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:12.439861 2556 patch_prober.go:28] interesting pod/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 15 12:50:12.439930 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:12.439900 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" podUID="cdbc4a2b-d869-48fe-834e-2233cfbe840d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 15 12:50:13.418783 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:13.418748 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" Jul 15 12:50:13.439602 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:13.439575 2556 patch_prober.go:28] interesting pod/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 15 12:50:13.440016 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:13.439624 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" podUID="cdbc4a2b-d869-48fe-834e-2233cfbe840d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 15 12:50:14.439376 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:14.439346 2556 patch_prober.go:28] interesting pod/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 15 12:50:14.439569 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:14.439397 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" podUID="cdbc4a2b-d869-48fe-834e-2233cfbe840d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 15 12:50:15.439662 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:15.439611 2556 patch_prober.go:28] interesting pod/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 15 12:50:15.439662 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:15.439665 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" podUID="cdbc4a2b-d869-48fe-834e-2233cfbe840d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 15 12:50:16.443141 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:16.443111 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:50:16.443563 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:16.443371 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:50:16.444266 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:16.444250 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:50:16.585584 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:16.585551 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:16.691658 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:16.691623 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:19.670278 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:19.670248 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:24.749833 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:24.749802 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:24.850733 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:50:24.850702 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:51:04.378992 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:04.378953 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:51:13.998042 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:13.998006 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:51:16.874266 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:16.874239 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:51:16.880545 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:16.880524 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:51:21.844237 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:21.844207 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:51:22.721059 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:22.720997 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:51:23.322937 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:23.322905 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:51:23.377070 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:23.377034 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:51:33.231618 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.231582 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq"] Jul 15 12:51:33.232227 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.231858 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" podUID="18185de2-f20c-4783-b04a-e32a96d74117" containerName="maas-api" containerID="cri-o://371ba374b311f9096aef2b67570dbec4e70425ed7bc8444f5a4efb4283ead6e3" gracePeriod=30 Jul 15 12:51:33.269552 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.269518 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:51:33.676121 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.676097 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" Jul 15 12:51:33.784814 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.784696 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18185de2-f20c-4783-b04a-e32a96d74117-maas-api-tls\") pod \"18185de2-f20c-4783-b04a-e32a96d74117\" (UID: \"18185de2-f20c-4783-b04a-e32a96d74117\") " Jul 15 12:51:33.784814 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.784739 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt6nw\" (UniqueName: \"kubernetes.io/projected/18185de2-f20c-4783-b04a-e32a96d74117-kube-api-access-bt6nw\") pod \"18185de2-f20c-4783-b04a-e32a96d74117\" (UID: \"18185de2-f20c-4783-b04a-e32a96d74117\") " Jul 15 12:51:33.785048 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.784876 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18185de2-f20c-4783-b04a-e32a96d74117-openshift-service-ca\") pod \"18185de2-f20c-4783-b04a-e32a96d74117\" (UID: \"18185de2-f20c-4783-b04a-e32a96d74117\") " Jul 15 12:51:33.785229 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.785205 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18185de2-f20c-4783-b04a-e32a96d74117-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "18185de2-f20c-4783-b04a-e32a96d74117" (UID: "18185de2-f20c-4783-b04a-e32a96d74117"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:51:33.786915 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.786897 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18185de2-f20c-4783-b04a-e32a96d74117-kube-api-access-bt6nw" (OuterVolumeSpecName: "kube-api-access-bt6nw") pod "18185de2-f20c-4783-b04a-e32a96d74117" (UID: "18185de2-f20c-4783-b04a-e32a96d74117"). InnerVolumeSpecName "kube-api-access-bt6nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:51:33.787004 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.786984 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18185de2-f20c-4783-b04a-e32a96d74117-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "18185de2-f20c-4783-b04a-e32a96d74117" (UID: "18185de2-f20c-4783-b04a-e32a96d74117"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:51:33.803087 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.803056 2556 generic.go:356] "Generic (PLEG): container finished" podID="18185de2-f20c-4783-b04a-e32a96d74117" containerID="371ba374b311f9096aef2b67570dbec4e70425ed7bc8444f5a4efb4283ead6e3" exitCode=0 Jul 15 12:51:33.803204 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.803101 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" Jul 15 12:51:33.803204 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.803106 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" event={"ID":"18185de2-f20c-4783-b04a-e32a96d74117","Type":"ContainerDied","Data":"371ba374b311f9096aef2b67570dbec4e70425ed7bc8444f5a4efb4283ead6e3"} Jul 15 12:51:33.803204 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.803151 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq" event={"ID":"18185de2-f20c-4783-b04a-e32a96d74117","Type":"ContainerDied","Data":"76f8f5c5c93be15c3b368bdb1f473b587beb4ce6e93470b95cfa4026dc66afa9"} Jul 15 12:51:33.803204 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.803174 2556 scope.go:117] "RemoveContainer" containerID="371ba374b311f9096aef2b67570dbec4e70425ed7bc8444f5a4efb4283ead6e3" Jul 15 12:51:33.812610 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.812596 2556 scope.go:117] "RemoveContainer" containerID="371ba374b311f9096aef2b67570dbec4e70425ed7bc8444f5a4efb4283ead6e3" Jul 15 12:51:33.812867 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:51:33.812848 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"371ba374b311f9096aef2b67570dbec4e70425ed7bc8444f5a4efb4283ead6e3\": container with ID starting with 371ba374b311f9096aef2b67570dbec4e70425ed7bc8444f5a4efb4283ead6e3 not found: ID does not exist" containerID="371ba374b311f9096aef2b67570dbec4e70425ed7bc8444f5a4efb4283ead6e3" Jul 15 12:51:33.812925 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.812874 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"371ba374b311f9096aef2b67570dbec4e70425ed7bc8444f5a4efb4283ead6e3"} err="failed to get container status \"371ba374b311f9096aef2b67570dbec4e70425ed7bc8444f5a4efb4283ead6e3\": rpc error: code = NotFound desc = could not find container \"371ba374b311f9096aef2b67570dbec4e70425ed7bc8444f5a4efb4283ead6e3\": container with ID starting with 371ba374b311f9096aef2b67570dbec4e70425ed7bc8444f5a4efb4283ead6e3 not found: ID does not exist" Jul 15 12:51:33.824001 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.823967 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq"] Jul 15 12:51:33.827711 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.827689 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-99c883-8cdd4b64f-llzjq"] Jul 15 12:51:33.885693 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.885673 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18185de2-f20c-4783-b04a-e32a96d74117-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:51:33.885693 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.885694 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18185de2-f20c-4783-b04a-e32a96d74117-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:51:33.885893 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:33.885703 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt6nw\" (UniqueName: \"kubernetes.io/projected/18185de2-f20c-4783-b04a-e32a96d74117-kube-api-access-bt6nw\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:51:34.854459 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:34.854427 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18185de2-f20c-4783-b04a-e32a96d74117" path="/var/lib/kubelet/pods/18185de2-f20c-4783-b04a-e32a96d74117/volumes" Jul 15 12:51:37.961738 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:37.961654 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:51:43.716833 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:43.716796 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:51:43.775024 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:43.774992 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:51:48.619952 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:48.619911 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t"] Jul 15 12:51:48.620365 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:48.620126 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" podUID="cdbc4a2b-d869-48fe-834e-2233cfbe840d" containerName="istio-proxy" containerID="cri-o://4a3b76a0f2f6e1b335fb3b7f23df7656435146294ffa66b0bd241ad5953e83a0" gracePeriod=30 Jul 15 12:51:48.640575 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:48.640543 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:51:49.946604 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:49.946570 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:51:50.046811 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:50.046764 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:51:53.868086 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:53.868060 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:51:53.892367 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:53.892296 2556 generic.go:356] "Generic (PLEG): container finished" podID="cdbc4a2b-d869-48fe-834e-2233cfbe840d" containerID="4a3b76a0f2f6e1b335fb3b7f23df7656435146294ffa66b0bd241ad5953e83a0" exitCode=0 Jul 15 12:51:53.892367 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:53.892350 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" Jul 15 12:51:53.892566 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:53.892377 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" event={"ID":"cdbc4a2b-d869-48fe-834e-2233cfbe840d","Type":"ContainerDied","Data":"4a3b76a0f2f6e1b335fb3b7f23df7656435146294ffa66b0bd241ad5953e83a0"} Jul 15 12:51:53.892566 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:53.892429 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t" event={"ID":"cdbc4a2b-d869-48fe-834e-2233cfbe840d","Type":"ContainerDied","Data":"eeeb7debdc4d070bcf8a83d98bcbb0dfbbc1e4c39acc38e5a402c23a643057f8"} Jul 15 12:51:53.892566 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:53.892448 2556 scope.go:117] "RemoveContainer" containerID="4a3b76a0f2f6e1b335fb3b7f23df7656435146294ffa66b0bd241ad5953e83a0" Jul 15 12:51:53.900863 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:53.900848 2556 scope.go:117] "RemoveContainer" containerID="cfb063102a6f7310ea22ce62fb94555090dc9d3357142fe7744569ad3d36c8c9" Jul 15 12:51:53.908887 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:53.908866 2556 scope.go:117] "RemoveContainer" containerID="4a3b76a0f2f6e1b335fb3b7f23df7656435146294ffa66b0bd241ad5953e83a0" Jul 15 12:51:53.909146 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:51:53.909129 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3b76a0f2f6e1b335fb3b7f23df7656435146294ffa66b0bd241ad5953e83a0\": container with ID starting with 4a3b76a0f2f6e1b335fb3b7f23df7656435146294ffa66b0bd241ad5953e83a0 not found: ID does not exist" containerID="4a3b76a0f2f6e1b335fb3b7f23df7656435146294ffa66b0bd241ad5953e83a0" Jul 15 12:51:53.909216 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:53.909154 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3b76a0f2f6e1b335fb3b7f23df7656435146294ffa66b0bd241ad5953e83a0"} err="failed to get container status \"4a3b76a0f2f6e1b335fb3b7f23df7656435146294ffa66b0bd241ad5953e83a0\": rpc error: code = NotFound desc = could not find container \"4a3b76a0f2f6e1b335fb3b7f23df7656435146294ffa66b0bd241ad5953e83a0\": container with ID starting with 4a3b76a0f2f6e1b335fb3b7f23df7656435146294ffa66b0bd241ad5953e83a0 not found: ID does not exist" Jul 15 12:51:53.909216 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:53.909174 2556 scope.go:117] "RemoveContainer" containerID="cfb063102a6f7310ea22ce62fb94555090dc9d3357142fe7744569ad3d36c8c9" Jul 15 12:51:53.909417 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:51:53.909401 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfb063102a6f7310ea22ce62fb94555090dc9d3357142fe7744569ad3d36c8c9\": container with ID starting with cfb063102a6f7310ea22ce62fb94555090dc9d3357142fe7744569ad3d36c8c9 not found: ID does not exist" containerID="cfb063102a6f7310ea22ce62fb94555090dc9d3357142fe7744569ad3d36c8c9" Jul 15 12:51:53.909475 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:53.909423 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb063102a6f7310ea22ce62fb94555090dc9d3357142fe7744569ad3d36c8c9"} err="failed to get container status \"cfb063102a6f7310ea22ce62fb94555090dc9d3357142fe7744569ad3d36c8c9\": rpc error: code = NotFound desc = could not find container \"cfb063102a6f7310ea22ce62fb94555090dc9d3357142fe7744569ad3d36c8c9\": container with ID starting with cfb063102a6f7310ea22ce62fb94555090dc9d3357142fe7744569ad3d36c8c9 not found: ID does not exist" Jul 15 12:51:54.041607 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.041580 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-workload-certs\") pod \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " Jul 15 12:51:54.041805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.041619 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-podinfo\") pod \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " Jul 15 12:51:54.041805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.041637 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-token\") pod \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " Jul 15 12:51:54.041805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.041660 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istiod-ca-cert\") pod \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " Jul 15 12:51:54.041805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.041694 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-credential-socket\") pod \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " Jul 15 12:51:54.041805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.041722 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-workload-socket\") pod \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " Jul 15 12:51:54.041805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.041764 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8f5bl\" (UniqueName: \"kubernetes.io/projected/cdbc4a2b-d869-48fe-834e-2233cfbe840d-kube-api-access-8f5bl\") pod \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " Jul 15 12:51:54.042152 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.041809 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-envoy\") pod \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " Jul 15 12:51:54.042152 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.041890 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cdbc4a2b-d869-48fe-834e-2233cfbe840d" (UID: "cdbc4a2b-d869-48fe-834e-2233cfbe840d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:51:54.042152 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.041932 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-data\") pod \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\" (UID: \"cdbc4a2b-d869-48fe-834e-2233cfbe840d\") " Jul 15 12:51:54.042152 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.042140 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-workload-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:51:54.042363 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.042323 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cdbc4a2b-d869-48fe-834e-2233cfbe840d" (UID: "cdbc4a2b-d869-48fe-834e-2233cfbe840d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:51:54.042420 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.042386 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cdbc4a2b-d869-48fe-834e-2233cfbe840d" (UID: "cdbc4a2b-d869-48fe-834e-2233cfbe840d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:51:54.042517 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.042497 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cdbc4a2b-d869-48fe-834e-2233cfbe840d" (UID: "cdbc4a2b-d869-48fe-834e-2233cfbe840d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:51:54.042885 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.042867 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-data" (OuterVolumeSpecName: "istio-data") pod "cdbc4a2b-d869-48fe-834e-2233cfbe840d" (UID: "cdbc4a2b-d869-48fe-834e-2233cfbe840d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:51:54.044557 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.044528 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cdbc4a2b-d869-48fe-834e-2233cfbe840d" (UID: "cdbc4a2b-d869-48fe-834e-2233cfbe840d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:51:54.044673 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.044598 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cdbc4a2b-d869-48fe-834e-2233cfbe840d" (UID: "cdbc4a2b-d869-48fe-834e-2233cfbe840d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:51:54.044673 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.044609 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-token" (OuterVolumeSpecName: "istio-token") pod "cdbc4a2b-d869-48fe-834e-2233cfbe840d" (UID: "cdbc4a2b-d869-48fe-834e-2233cfbe840d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:51:54.044673 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.044632 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdbc4a2b-d869-48fe-834e-2233cfbe840d-kube-api-access-8f5bl" (OuterVolumeSpecName: "kube-api-access-8f5bl") pod "cdbc4a2b-d869-48fe-834e-2233cfbe840d" (UID: "cdbc4a2b-d869-48fe-834e-2233cfbe840d"). InnerVolumeSpecName "kube-api-access-8f5bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:51:54.142504 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.142439 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-envoy\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:51:54.142504 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.142460 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-data\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:51:54.142504 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.142468 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-podinfo\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:51:54.142504 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.142476 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istio-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:51:54.142504 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.142484 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdbc4a2b-d869-48fe-834e-2233cfbe840d-istiod-ca-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:51:54.142504 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.142491 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-credential-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:51:54.142504 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.142499 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbc4a2b-d869-48fe-834e-2233cfbe840d-workload-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:51:54.142504 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.142507 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f5bl\" (UniqueName: \"kubernetes.io/projected/cdbc4a2b-d869-48fe-834e-2233cfbe840d-kube-api-access-8f5bl\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:51:54.215378 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.215347 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t"] Jul 15 12:51:54.219209 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.219186 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-99c883-openshift-default-76dd5f9757-k576t"] Jul 15 12:51:54.856254 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:54.856220 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdbc4a2b-d869-48fe-834e-2233cfbe840d" path="/var/lib/kubelet/pods/cdbc4a2b-d869-48fe-834e-2233cfbe840d/volumes" Jul 15 12:51:59.334108 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.334076 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k"] Jul 15 12:51:59.334698 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.334461 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" podUID="956499ca-1933-4efa-b5e0-0744799baff6" containerName="maas-api" containerID="cri-o://e7a2d5e4f84b37a2f3096be03f4aa5424e4fcc4cdc3137e8cec6f3b21a86ce31" gracePeriod=30 Jul 15 12:51:59.386670 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.386639 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:51:59.585911 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.585860 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" Jul 15 12:51:59.679026 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.678998 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8sgl\" (UniqueName: \"kubernetes.io/projected/956499ca-1933-4efa-b5e0-0744799baff6-kube-api-access-l8sgl\") pod \"956499ca-1933-4efa-b5e0-0744799baff6\" (UID: \"956499ca-1933-4efa-b5e0-0744799baff6\") " Jul 15 12:51:59.679208 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.679059 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/956499ca-1933-4efa-b5e0-0744799baff6-maas-api-tls\") pod \"956499ca-1933-4efa-b5e0-0744799baff6\" (UID: \"956499ca-1933-4efa-b5e0-0744799baff6\") " Jul 15 12:51:59.679208 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.679090 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/956499ca-1933-4efa-b5e0-0744799baff6-openshift-service-ca\") pod \"956499ca-1933-4efa-b5e0-0744799baff6\" (UID: \"956499ca-1933-4efa-b5e0-0744799baff6\") " Jul 15 12:51:59.679481 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.679458 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956499ca-1933-4efa-b5e0-0744799baff6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "956499ca-1933-4efa-b5e0-0744799baff6" (UID: "956499ca-1933-4efa-b5e0-0744799baff6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:51:59.681354 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.681321 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956499ca-1933-4efa-b5e0-0744799baff6-kube-api-access-l8sgl" (OuterVolumeSpecName: "kube-api-access-l8sgl") pod "956499ca-1933-4efa-b5e0-0744799baff6" (UID: "956499ca-1933-4efa-b5e0-0744799baff6"). InnerVolumeSpecName "kube-api-access-l8sgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:51:59.681462 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.681356 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/956499ca-1933-4efa-b5e0-0744799baff6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "956499ca-1933-4efa-b5e0-0744799baff6" (UID: "956499ca-1933-4efa-b5e0-0744799baff6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:51:59.779708 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.779661 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/956499ca-1933-4efa-b5e0-0744799baff6-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:51:59.779708 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.779700 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/956499ca-1933-4efa-b5e0-0744799baff6-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:51:59.779708 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.779710 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8sgl\" (UniqueName: \"kubernetes.io/projected/956499ca-1933-4efa-b5e0-0744799baff6-kube-api-access-l8sgl\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:51:59.919091 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.919000 2556 generic.go:356] "Generic (PLEG): container finished" podID="956499ca-1933-4efa-b5e0-0744799baff6" containerID="e7a2d5e4f84b37a2f3096be03f4aa5424e4fcc4cdc3137e8cec6f3b21a86ce31" exitCode=0 Jul 15 12:51:59.919091 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.919046 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" Jul 15 12:51:59.919274 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.919088 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" event={"ID":"956499ca-1933-4efa-b5e0-0744799baff6","Type":"ContainerDied","Data":"e7a2d5e4f84b37a2f3096be03f4aa5424e4fcc4cdc3137e8cec6f3b21a86ce31"} Jul 15 12:51:59.919274 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.919144 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k" event={"ID":"956499ca-1933-4efa-b5e0-0744799baff6","Type":"ContainerDied","Data":"8904bf07cb5c8989b8c534fe00b8c5f1047c9bd47ac389601cab9e4c2c5f8b21"} Jul 15 12:51:59.919274 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.919166 2556 scope.go:117] "RemoveContainer" containerID="e7a2d5e4f84b37a2f3096be03f4aa5424e4fcc4cdc3137e8cec6f3b21a86ce31" Jul 15 12:51:59.927937 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.927919 2556 scope.go:117] "RemoveContainer" containerID="e7a2d5e4f84b37a2f3096be03f4aa5424e4fcc4cdc3137e8cec6f3b21a86ce31" Jul 15 12:51:59.928185 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:51:59.928167 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a2d5e4f84b37a2f3096be03f4aa5424e4fcc4cdc3137e8cec6f3b21a86ce31\": container with ID starting with e7a2d5e4f84b37a2f3096be03f4aa5424e4fcc4cdc3137e8cec6f3b21a86ce31 not found: ID does not exist" containerID="e7a2d5e4f84b37a2f3096be03f4aa5424e4fcc4cdc3137e8cec6f3b21a86ce31" Jul 15 12:51:59.928239 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.928193 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a2d5e4f84b37a2f3096be03f4aa5424e4fcc4cdc3137e8cec6f3b21a86ce31"} err="failed to get container status \"e7a2d5e4f84b37a2f3096be03f4aa5424e4fcc4cdc3137e8cec6f3b21a86ce31\": rpc error: code = NotFound desc = could not find container \"e7a2d5e4f84b37a2f3096be03f4aa5424e4fcc4cdc3137e8cec6f3b21a86ce31\": container with ID starting with e7a2d5e4f84b37a2f3096be03f4aa5424e4fcc4cdc3137e8cec6f3b21a86ce31 not found: ID does not exist" Jul 15 12:51:59.939229 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.939208 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k"] Jul 15 12:51:59.942877 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:51:59.942857 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0eb9b9-7d57d5d4b9-fdg4k"] Jul 15 12:52:00.853450 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:00.853403 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="956499ca-1933-4efa-b5e0-0744799baff6" path="/var/lib/kubelet/pods/956499ca-1933-4efa-b5e0-0744799baff6/volumes" Jul 15 12:52:04.069431 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:04.069404 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:09.670820 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:09.670786 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:09.771070 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:09.771004 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:15.681886 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.681847 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp"] Jul 15 12:52:15.682328 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.682158 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdbc4a2b-d869-48fe-834e-2233cfbe840d" containerName="istio-proxy" Jul 15 12:52:15.682328 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.682169 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdbc4a2b-d869-48fe-834e-2233cfbe840d" containerName="istio-proxy" Jul 15 12:52:15.682328 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.682186 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdbc4a2b-d869-48fe-834e-2233cfbe840d" containerName="istio-proxy" Jul 15 12:52:15.682328 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.682192 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdbc4a2b-d869-48fe-834e-2233cfbe840d" containerName="istio-proxy" Jul 15 12:52:15.682328 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.682214 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18185de2-f20c-4783-b04a-e32a96d74117" containerName="maas-api" Jul 15 12:52:15.682328 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.682223 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="18185de2-f20c-4783-b04a-e32a96d74117" containerName="maas-api" Jul 15 12:52:15.682328 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.682234 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="956499ca-1933-4efa-b5e0-0744799baff6" containerName="maas-api" Jul 15 12:52:15.682328 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.682240 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="956499ca-1933-4efa-b5e0-0744799baff6" containerName="maas-api" Jul 15 12:52:15.682328 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.682283 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdbc4a2b-d869-48fe-834e-2233cfbe840d" containerName="istio-proxy" Jul 15 12:52:15.682328 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.682294 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="956499ca-1933-4efa-b5e0-0744799baff6" containerName="maas-api" Jul 15 12:52:15.682328 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.682305 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="18185de2-f20c-4783-b04a-e32a96d74117" containerName="maas-api" Jul 15 12:52:15.682328 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.682311 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdbc4a2b-d869-48fe-834e-2233cfbe840d" containerName="istio-proxy" Jul 15 12:52:15.686705 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.686679 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.689747 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.689722 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-20d2f8-openshift-default-dockercfg-xbvzk\"" Jul 15 12:52:15.696411 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.696389 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp"] Jul 15 12:52:15.802659 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.802626 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs7cs\" (UniqueName: \"kubernetes.io/projected/2fa83c7b-db09-4266-8581-7292a12dd17d-kube-api-access-hs7cs\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.802851 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.802668 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-data\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.802851 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.802693 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-token\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.802851 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.802825 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-podinfo\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.802963 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.802855 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-workload-socket\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.802963 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.802871 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-workload-certs\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.802963 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.802937 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-credential-socket\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.803061 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.802964 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa83c7b-db09-4266-8581-7292a12dd17d-istiod-ca-cert\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.803061 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.802993 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-envoy\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.884696 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.884662 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:15.904384 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.904358 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs7cs\" (UniqueName: \"kubernetes.io/projected/2fa83c7b-db09-4266-8581-7292a12dd17d-kube-api-access-hs7cs\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.904533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.904400 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-data\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.904533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.904418 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-token\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.904533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.904443 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-podinfo\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.904533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.904460 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-workload-socket\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.904748 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.904588 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-workload-certs\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.904748 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.904646 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-credential-socket\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.904748 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.904672 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa83c7b-db09-4266-8581-7292a12dd17d-istiod-ca-cert\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.904919 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.904752 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-envoy\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.904919 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.904877 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-workload-socket\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.905027 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.904946 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-data\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.905027 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.904984 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-workload-certs\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.905027 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.904998 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-credential-socket\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.905447 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.905424 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa83c7b-db09-4266-8581-7292a12dd17d-istiod-ca-cert\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.907348 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.907327 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-envoy\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.907457 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.907430 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-podinfo\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.917335 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.917314 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-token\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.917856 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.917835 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs7cs\" (UniqueName: \"kubernetes.io/projected/2fa83c7b-db09-4266-8581-7292a12dd17d-kube-api-access-hs7cs\") pod \"e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:15.985720 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.985630 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:15.998219 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:15.998184 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:16.040424 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:16.040383 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:16.132260 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:16.132178 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp"] Jul 15 12:52:16.134407 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:52:16.134379 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa83c7b_db09_4266_8581_7292a12dd17d.slice/crio-61e09b973aca7ec3637b6faf749519215b2175656ee9689ab11eb360f91ed5e7 WatchSource:0}: Error finding container 61e09b973aca7ec3637b6faf749519215b2175656ee9689ab11eb360f91ed5e7: Status 404 returned error can't find the container with id 61e09b973aca7ec3637b6faf749519215b2175656ee9689ab11eb360f91ed5e7 Jul 15 12:52:16.136545 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:16.136511 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:52:16.136604 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:16.136578 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:52:16.136640 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:16.136604 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:52:16.993971 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:16.993875 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" event={"ID":"2fa83c7b-db09-4266-8581-7292a12dd17d","Type":"ContainerStarted","Data":"80e7e1a79c10e4aff953ce2fa5099d3ebf0d7915f60548f4978188423bd54782"} Jul 15 12:52:16.993971 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:16.993940 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" event={"ID":"2fa83c7b-db09-4266-8581-7292a12dd17d","Type":"ContainerStarted","Data":"61e09b973aca7ec3637b6faf749519215b2175656ee9689ab11eb360f91ed5e7"} Jul 15 12:52:16.998552 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:16.998516 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:17.015850 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:17.015769 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" podStartSLOduration=2.015748816 podStartE2EDuration="2.015748816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:52:17.01543904 +0000 UTC m=+2460.649833988" watchObservedRunningTime="2026-07-15 12:52:17.015748816 +0000 UTC m=+2460.650143765" Jul 15 12:52:18.004815 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:18.004760 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:18.945382 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:18.945349 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:19.002120 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.002085 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:19.003381 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.003352 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:19.005275 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.005253 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds"] Jul 15 12:52:19.009843 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.009825 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" Jul 15 12:52:19.012632 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.012613 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-20d2f8-serving-cert\"" Jul 15 12:52:19.012728 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.012675 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 12:52:19.012728 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.012720 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bnnf9\"" Jul 15 12:52:19.012935 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.012919 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 12:52:19.018954 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.018929 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds"] Jul 15 12:52:19.021167 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.021145 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:19.104586 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.104554 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:19.132038 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.132004 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9f6330-6eaf-4bb1-8a93-76b643775026-maas-api-tls\") pod \"maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds\" (UID: \"0a9f6330-6eaf-4bb1-8a93-76b643775026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" Jul 15 12:52:19.132171 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.132081 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4wjd\" (UniqueName: \"kubernetes.io/projected/0a9f6330-6eaf-4bb1-8a93-76b643775026-kube-api-access-r4wjd\") pod \"maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds\" (UID: \"0a9f6330-6eaf-4bb1-8a93-76b643775026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" Jul 15 12:52:19.132171 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.132120 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9f6330-6eaf-4bb1-8a93-76b643775026-openshift-service-ca\") pod \"maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds\" (UID: \"0a9f6330-6eaf-4bb1-8a93-76b643775026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" Jul 15 12:52:19.233096 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.233004 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9f6330-6eaf-4bb1-8a93-76b643775026-maas-api-tls\") pod \"maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds\" (UID: \"0a9f6330-6eaf-4bb1-8a93-76b643775026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" Jul 15 12:52:19.233096 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.233059 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4wjd\" (UniqueName: \"kubernetes.io/projected/0a9f6330-6eaf-4bb1-8a93-76b643775026-kube-api-access-r4wjd\") pod \"maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds\" (UID: \"0a9f6330-6eaf-4bb1-8a93-76b643775026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" Jul 15 12:52:19.233096 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.233094 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9f6330-6eaf-4bb1-8a93-76b643775026-openshift-service-ca\") pod \"maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds\" (UID: \"0a9f6330-6eaf-4bb1-8a93-76b643775026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" Jul 15 12:52:19.233400 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:52:19.233157 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-serving-cert: secret "maas-api-e2e-api-a-20d2f8-serving-cert" not found Jul 15 12:52:19.233400 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:52:19.233227 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a9f6330-6eaf-4bb1-8a93-76b643775026-maas-api-tls podName:0a9f6330-6eaf-4bb1-8a93-76b643775026 nodeName:}" failed. No retries permitted until 2026-07-15 12:52:19.733211283 +0000 UTC m=+2463.367606210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a9f6330-6eaf-4bb1-8a93-76b643775026-maas-api-tls") pod "maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" (UID: "0a9f6330-6eaf-4bb1-8a93-76b643775026") : secret "maas-api-e2e-api-a-20d2f8-serving-cert" not found Jul 15 12:52:19.233767 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.233745 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9f6330-6eaf-4bb1-8a93-76b643775026-openshift-service-ca\") pod \"maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds\" (UID: \"0a9f6330-6eaf-4bb1-8a93-76b643775026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" Jul 15 12:52:19.247138 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.246900 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4wjd\" (UniqueName: \"kubernetes.io/projected/0a9f6330-6eaf-4bb1-8a93-76b643775026-kube-api-access-r4wjd\") pod \"maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds\" (UID: \"0a9f6330-6eaf-4bb1-8a93-76b643775026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" Jul 15 12:52:19.456669 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.456632 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:19.700747 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.700696 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:19.736667 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.736633 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9f6330-6eaf-4bb1-8a93-76b643775026-maas-api-tls\") pod \"maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds\" (UID: \"0a9f6330-6eaf-4bb1-8a93-76b643775026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" Jul 15 12:52:19.739122 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.739093 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9f6330-6eaf-4bb1-8a93-76b643775026-maas-api-tls\") pod \"maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds\" (UID: \"0a9f6330-6eaf-4bb1-8a93-76b643775026\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" Jul 15 12:52:19.805968 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.805925 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:19.920830 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:19.920799 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" Jul 15 12:52:20.051804 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:20.051758 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds"] Jul 15 12:52:20.052690 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:52:20.052666 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9f6330_6eaf_4bb1_8a93_76b643775026.slice/crio-3032e1a451eb9d973ef81ad95228c0946b0d2270ec4519a1bad1f816f544c2c7 WatchSource:0}: Error finding container 3032e1a451eb9d973ef81ad95228c0946b0d2270ec4519a1bad1f816f544c2c7: Status 404 returned error can't find the container with id 3032e1a451eb9d973ef81ad95228c0946b0d2270ec4519a1bad1f816f544c2c7 Jul 15 12:52:20.054425 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:20.054411 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 12:52:21.013289 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:21.013258 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" event={"ID":"0a9f6330-6eaf-4bb1-8a93-76b643775026","Type":"ContainerStarted","Data":"95e2e108d50bb7c691b2f1355ce5a783c10dc9113ca583b859af476f01c21e32"} Jul 15 12:52:21.013499 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:21.013318 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" event={"ID":"0a9f6330-6eaf-4bb1-8a93-76b643775026","Type":"ContainerStarted","Data":"3032e1a451eb9d973ef81ad95228c0946b0d2270ec4519a1bad1f816f544c2c7"} Jul 15 12:52:21.013499 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:21.013345 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" Jul 15 12:52:21.030584 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:21.030540 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" podStartSLOduration=2.543037543 podStartE2EDuration="3.030524509s" totalImagesPullingTime="487.486966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:52:21.028817022 +0000 UTC m=+2464.663211970" watchObservedRunningTime="2026-07-15 12:52:21.030524509 +0000 UTC m=+2464.664919533" Jul 15 12:52:21.052491 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:21.052449 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:21.151521 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:21.151494 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:23.460364 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:23.460329 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:23.933460 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:23.933429 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:24.242393 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:24.242304 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:24.685872 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:24.685831 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:27.024338 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.024309 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" Jul 15 12:52:27.131976 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.131938 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:27.205067 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.205035 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt"] Jul 15 12:52:27.216076 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.216050 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" Jul 15 12:52:27.220945 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.220918 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-44c6c5-serving-cert\"" Jul 15 12:52:27.222554 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.222484 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt"] Jul 15 12:52:27.222805 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.222752 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:27.282314 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.282240 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:27.319204 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.319170 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b344f5af-51f3-4ba6-9ca8-676ad0fa6c26-openshift-service-ca\") pod \"maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt\" (UID: \"b344f5af-51f3-4ba6-9ca8-676ad0fa6c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" Jul 15 12:52:27.319488 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.319468 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvj4t\" (UniqueName: \"kubernetes.io/projected/b344f5af-51f3-4ba6-9ca8-676ad0fa6c26-kube-api-access-cvj4t\") pod \"maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt\" (UID: \"b344f5af-51f3-4ba6-9ca8-676ad0fa6c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" Jul 15 12:52:27.319635 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.319622 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b344f5af-51f3-4ba6-9ca8-676ad0fa6c26-maas-api-tls\") pod \"maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt\" (UID: \"b344f5af-51f3-4ba6-9ca8-676ad0fa6c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" Jul 15 12:52:27.420408 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.420377 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b344f5af-51f3-4ba6-9ca8-676ad0fa6c26-openshift-service-ca\") pod \"maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt\" (UID: \"b344f5af-51f3-4ba6-9ca8-676ad0fa6c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" Jul 15 12:52:27.420609 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.420429 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvj4t\" (UniqueName: \"kubernetes.io/projected/b344f5af-51f3-4ba6-9ca8-676ad0fa6c26-kube-api-access-cvj4t\") pod \"maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt\" (UID: \"b344f5af-51f3-4ba6-9ca8-676ad0fa6c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" Jul 15 12:52:27.420609 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.420470 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b344f5af-51f3-4ba6-9ca8-676ad0fa6c26-maas-api-tls\") pod \"maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt\" (UID: \"b344f5af-51f3-4ba6-9ca8-676ad0fa6c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" Jul 15 12:52:27.421280 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.421252 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b344f5af-51f3-4ba6-9ca8-676ad0fa6c26-openshift-service-ca\") pod \"maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt\" (UID: \"b344f5af-51f3-4ba6-9ca8-676ad0fa6c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" Jul 15 12:52:27.423206 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.423184 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b344f5af-51f3-4ba6-9ca8-676ad0fa6c26-maas-api-tls\") pod \"maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt\" (UID: \"b344f5af-51f3-4ba6-9ca8-676ad0fa6c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" Jul 15 12:52:27.434243 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.434221 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvj4t\" (UniqueName: \"kubernetes.io/projected/b344f5af-51f3-4ba6-9ca8-676ad0fa6c26-kube-api-access-cvj4t\") pod \"maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt\" (UID: \"b344f5af-51f3-4ba6-9ca8-676ad0fa6c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" Jul 15 12:52:27.453260 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.453233 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:27.539842 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.539703 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" Jul 15 12:52:27.706800 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.702710 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt"] Jul 15 12:52:27.723865 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:52:27.723832 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb344f5af_51f3_4ba6_9ca8_676ad0fa6c26.slice/crio-f614bcd9b6025f22f5ae239b85d615471ec9b14edd26a4c8104f2cd6df80a95c WatchSource:0}: Error finding container f614bcd9b6025f22f5ae239b85d615471ec9b14edd26a4c8104f2cd6df80a95c: Status 404 returned error can't find the container with id f614bcd9b6025f22f5ae239b85d615471ec9b14edd26a4c8104f2cd6df80a95c Jul 15 12:52:27.829832 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:27.829705 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:28.011465 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:28.011432 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:28.045903 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:28.045870 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" event={"ID":"b344f5af-51f3-4ba6-9ca8-676ad0fa6c26","Type":"ContainerStarted","Data":"f614bcd9b6025f22f5ae239b85d615471ec9b14edd26a4c8104f2cd6df80a95c"} Jul 15 12:52:29.052066 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:29.052025 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" event={"ID":"b344f5af-51f3-4ba6-9ca8-676ad0fa6c26","Type":"ContainerStarted","Data":"9e86022411c4098786d680d6fdc44fce238b83fb9ad0b50af7054f76a5a8b42a"} Jul 15 12:52:29.052477 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:29.052162 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" Jul 15 12:52:29.069618 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:29.069570 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" podStartSLOduration=1.666731266 podStartE2EDuration="2.069556106s" totalImagesPullingTime="402.82484ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:52:29.068595347 +0000 UTC m=+2472.702990296" watchObservedRunningTime="2026-07-15 12:52:29.069556106 +0000 UTC m=+2472.703951053" Jul 15 12:52:31.639533 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:31.639493 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:32.072338 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:32.072302 2556 generic.go:356] "Generic (PLEG): container finished" podID="2fa83c7b-db09-4266-8581-7292a12dd17d" containerID="80e7e1a79c10e4aff953ce2fa5099d3ebf0d7915f60548f4978188423bd54782" exitCode=0 Jul 15 12:52:32.072571 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:32.072371 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" event={"ID":"2fa83c7b-db09-4266-8581-7292a12dd17d","Type":"ContainerDied","Data":"80e7e1a79c10e4aff953ce2fa5099d3ebf0d7915f60548f4978188423bd54782"} Jul 15 12:52:32.072814 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:32.072795 2556 scope.go:117] "RemoveContainer" containerID="80e7e1a79c10e4aff953ce2fa5099d3ebf0d7915f60548f4978188423bd54782" Jul 15 12:52:32.073804 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:32.073754 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:52:32.073909 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:32.073844 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:52:32.073909 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:32.073883 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:52:33.081994 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:33.081960 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" event={"ID":"2fa83c7b-db09-4266-8581-7292a12dd17d","Type":"ContainerStarted","Data":"c28c100ed60205e205498377a8e85082909333ca3d1109a5e1a2110f956e0919"} Jul 15 12:52:33.999236 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:33.999196 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:34.000440 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:34.000405 2556 patch_prober.go:28] interesting pod/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 15 12:52:34.000587 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:34.000466 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" podUID="2fa83c7b-db09-4266-8581-7292a12dd17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 15 12:52:34.002531 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:34.002503 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:34.999584 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:34.999544 2556 patch_prober.go:28] interesting pod/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 15 12:52:35.000069 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:34.999606 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" podUID="2fa83c7b-db09-4266-8581-7292a12dd17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 15 12:52:35.064683 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:35.064654 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" Jul 15 12:52:35.998980 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:35.998937 2556 patch_prober.go:28] interesting pod/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 15 12:52:35.999666 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:35.999242 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" podUID="2fa83c7b-db09-4266-8581-7292a12dd17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 15 12:52:36.174504 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:36.174467 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:36.999417 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:36.999370 2556 patch_prober.go:28] interesting pod/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 15 12:52:36.999628 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:36.999438 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" podUID="2fa83c7b-db09-4266-8581-7292a12dd17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 15 12:52:38.002849 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:38.002817 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:38.003858 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:38.003835 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:52:40.219722 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:40.219684 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:40.320879 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:40.320849 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:49.140028 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:49.139993 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:49.188412 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:49.188382 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:50.202282 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:50.202250 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:50.302013 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:50.301985 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:50.415156 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:50.415110 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds"] Jul 15 12:52:50.415811 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:50.415367 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" podUID="0a9f6330-6eaf-4bb1-8a93-76b643775026" containerName="maas-api" containerID="cri-o://95e2e108d50bb7c691b2f1355ce5a783c10dc9113ca583b859af476f01c21e32" gracePeriod=30 Jul 15 12:52:50.438067 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:50.438043 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:50.658395 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:50.658372 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" Jul 15 12:52:50.716151 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:50.716117 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9f6330-6eaf-4bb1-8a93-76b643775026-openshift-service-ca\") pod \"0a9f6330-6eaf-4bb1-8a93-76b643775026\" (UID: \"0a9f6330-6eaf-4bb1-8a93-76b643775026\") " Jul 15 12:52:50.716339 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:50.716186 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4wjd\" (UniqueName: \"kubernetes.io/projected/0a9f6330-6eaf-4bb1-8a93-76b643775026-kube-api-access-r4wjd\") pod \"0a9f6330-6eaf-4bb1-8a93-76b643775026\" (UID: \"0a9f6330-6eaf-4bb1-8a93-76b643775026\") " Jul 15 12:52:50.716527 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:50.716493 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9f6330-6eaf-4bb1-8a93-76b643775026-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a9f6330-6eaf-4bb1-8a93-76b643775026" (UID: "0a9f6330-6eaf-4bb1-8a93-76b643775026"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:52:50.718444 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:50.718420 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9f6330-6eaf-4bb1-8a93-76b643775026-kube-api-access-r4wjd" (OuterVolumeSpecName: "kube-api-access-r4wjd") pod "0a9f6330-6eaf-4bb1-8a93-76b643775026" (UID: "0a9f6330-6eaf-4bb1-8a93-76b643775026"). InnerVolumeSpecName "kube-api-access-r4wjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:52:50.816790 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:50.816708 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9f6330-6eaf-4bb1-8a93-76b643775026-maas-api-tls\") pod \"0a9f6330-6eaf-4bb1-8a93-76b643775026\" (UID: \"0a9f6330-6eaf-4bb1-8a93-76b643775026\") " Jul 15 12:52:50.816928 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:50.816916 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4wjd\" (UniqueName: \"kubernetes.io/projected/0a9f6330-6eaf-4bb1-8a93-76b643775026-kube-api-access-r4wjd\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:52:50.816974 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:50.816930 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9f6330-6eaf-4bb1-8a93-76b643775026-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:52:50.818791 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:50.818754 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a9f6330-6eaf-4bb1-8a93-76b643775026-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a9f6330-6eaf-4bb1-8a93-76b643775026" (UID: "0a9f6330-6eaf-4bb1-8a93-76b643775026"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:52:50.917635 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:50.917604 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9f6330-6eaf-4bb1-8a93-76b643775026-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:52:51.176348 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:51.176261 2556 generic.go:356] "Generic (PLEG): container finished" podID="0a9f6330-6eaf-4bb1-8a93-76b643775026" containerID="95e2e108d50bb7c691b2f1355ce5a783c10dc9113ca583b859af476f01c21e32" exitCode=0 Jul 15 12:52:51.176348 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:51.176306 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" Jul 15 12:52:51.176348 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:51.176331 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" event={"ID":"0a9f6330-6eaf-4bb1-8a93-76b643775026","Type":"ContainerDied","Data":"95e2e108d50bb7c691b2f1355ce5a783c10dc9113ca583b859af476f01c21e32"} Jul 15 12:52:51.176591 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:51.176373 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds" event={"ID":"0a9f6330-6eaf-4bb1-8a93-76b643775026","Type":"ContainerDied","Data":"3032e1a451eb9d973ef81ad95228c0946b0d2270ec4519a1bad1f816f544c2c7"} Jul 15 12:52:51.176591 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:51.176395 2556 scope.go:117] "RemoveContainer" containerID="95e2e108d50bb7c691b2f1355ce5a783c10dc9113ca583b859af476f01c21e32" Jul 15 12:52:51.184728 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:51.184708 2556 scope.go:117] "RemoveContainer" containerID="95e2e108d50bb7c691b2f1355ce5a783c10dc9113ca583b859af476f01c21e32" Jul 15 12:52:51.185039 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:52:51.185018 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e2e108d50bb7c691b2f1355ce5a783c10dc9113ca583b859af476f01c21e32\": container with ID starting with 95e2e108d50bb7c691b2f1355ce5a783c10dc9113ca583b859af476f01c21e32 not found: ID does not exist" containerID="95e2e108d50bb7c691b2f1355ce5a783c10dc9113ca583b859af476f01c21e32" Jul 15 12:52:51.185117 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:51.185051 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e2e108d50bb7c691b2f1355ce5a783c10dc9113ca583b859af476f01c21e32"} err="failed to get container status \"95e2e108d50bb7c691b2f1355ce5a783c10dc9113ca583b859af476f01c21e32\": rpc error: code = NotFound desc = could not find container \"95e2e108d50bb7c691b2f1355ce5a783c10dc9113ca583b859af476f01c21e32\": container with ID starting with 95e2e108d50bb7c691b2f1355ce5a783c10dc9113ca583b859af476f01c21e32 not found: ID does not exist" Jul 15 12:52:51.193184 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:51.193159 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds"] Jul 15 12:52:51.196681 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:51.196659 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-20d2f8-676fbfb8d6-7dgds"] Jul 15 12:52:52.854392 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:52.854354 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a9f6330-6eaf-4bb1-8a93-76b643775026" path="/var/lib/kubelet/pods/0a9f6330-6eaf-4bb1-8a93-76b643775026/volumes" Jul 15 12:52:54.793933 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:52:54.793903 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:00.407158 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:00.407128 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:00.459570 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:00.459533 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:05.441721 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:05.441687 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp"] Jul 15 12:53:05.442227 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:05.441931 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" podUID="2fa83c7b-db09-4266-8581-7292a12dd17d" containerName="istio-proxy" containerID="cri-o://c28c100ed60205e205498377a8e85082909333ca3d1109a5e1a2110f956e0919" gracePeriod=30 Jul 15 12:53:05.456137 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:05.456102 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:05.692795 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:05.692677 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:06.299230 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:06.299197 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:06.399178 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:06.399145 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:08.005351 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:08.005263 2556 patch_prober.go:28] interesting pod/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 12:53:08.005351 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:08.005324 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" podUID="2fa83c7b-db09-4266-8581-7292a12dd17d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 12:53:10.688757 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.688733 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:53:10.774648 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.774621 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-workload-socket\") pod \"2fa83c7b-db09-4266-8581-7292a12dd17d\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " Jul 15 12:53:10.774828 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.774662 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs7cs\" (UniqueName: \"kubernetes.io/projected/2fa83c7b-db09-4266-8581-7292a12dd17d-kube-api-access-hs7cs\") pod \"2fa83c7b-db09-4266-8581-7292a12dd17d\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " Jul 15 12:53:10.774828 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.774689 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-credential-socket\") pod \"2fa83c7b-db09-4266-8581-7292a12dd17d\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " Jul 15 12:53:10.774828 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.774707 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-data\") pod \"2fa83c7b-db09-4266-8581-7292a12dd17d\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " Jul 15 12:53:10.774828 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.774722 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa83c7b-db09-4266-8581-7292a12dd17d-istiod-ca-cert\") pod \"2fa83c7b-db09-4266-8581-7292a12dd17d\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " Jul 15 12:53:10.774828 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.774745 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-podinfo\") pod \"2fa83c7b-db09-4266-8581-7292a12dd17d\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " Jul 15 12:53:10.774828 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.774760 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-envoy\") pod \"2fa83c7b-db09-4266-8581-7292a12dd17d\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " Jul 15 12:53:10.774828 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.774812 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-token\") pod \"2fa83c7b-db09-4266-8581-7292a12dd17d\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " Jul 15 12:53:10.775162 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.774870 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-workload-certs\") pod \"2fa83c7b-db09-4266-8581-7292a12dd17d\" (UID: \"2fa83c7b-db09-4266-8581-7292a12dd17d\") " Jul 15 12:53:10.775162 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.774962 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2fa83c7b-db09-4266-8581-7292a12dd17d" (UID: "2fa83c7b-db09-4266-8581-7292a12dd17d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:53:10.775162 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.774997 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2fa83c7b-db09-4266-8581-7292a12dd17d" (UID: "2fa83c7b-db09-4266-8581-7292a12dd17d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:53:10.775322 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.775173 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-workload-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:53:10.775322 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.775181 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2fa83c7b-db09-4266-8581-7292a12dd17d" (UID: "2fa83c7b-db09-4266-8581-7292a12dd17d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:53:10.775322 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.775188 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-credential-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:53:10.775474 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.775357 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa83c7b-db09-4266-8581-7292a12dd17d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2fa83c7b-db09-4266-8581-7292a12dd17d" (UID: "2fa83c7b-db09-4266-8581-7292a12dd17d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:53:10.775935 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.775879 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-data" (OuterVolumeSpecName: "istio-data") pod "2fa83c7b-db09-4266-8581-7292a12dd17d" (UID: "2fa83c7b-db09-4266-8581-7292a12dd17d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:53:10.777134 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.777102 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa83c7b-db09-4266-8581-7292a12dd17d-kube-api-access-hs7cs" (OuterVolumeSpecName: "kube-api-access-hs7cs") pod "2fa83c7b-db09-4266-8581-7292a12dd17d" (UID: "2fa83c7b-db09-4266-8581-7292a12dd17d"). InnerVolumeSpecName "kube-api-access-hs7cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:53:10.777249 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.777230 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2fa83c7b-db09-4266-8581-7292a12dd17d" (UID: "2fa83c7b-db09-4266-8581-7292a12dd17d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:53:10.777703 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.777676 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2fa83c7b-db09-4266-8581-7292a12dd17d" (UID: "2fa83c7b-db09-4266-8581-7292a12dd17d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:53:10.777748 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.777696 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-token" (OuterVolumeSpecName: "istio-token") pod "2fa83c7b-db09-4266-8581-7292a12dd17d" (UID: "2fa83c7b-db09-4266-8581-7292a12dd17d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:53:10.876413 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.876388 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs7cs\" (UniqueName: \"kubernetes.io/projected/2fa83c7b-db09-4266-8581-7292a12dd17d-kube-api-access-hs7cs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:53:10.876413 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.876411 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-data\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:53:10.876551 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.876420 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa83c7b-db09-4266-8581-7292a12dd17d-istiod-ca-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:53:10.876551 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.876427 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-podinfo\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:53:10.876551 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.876435 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-envoy\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:53:10.876551 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.876442 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa83c7b-db09-4266-8581-7292a12dd17d-istio-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:53:10.876551 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:10.876448 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa83c7b-db09-4266-8581-7292a12dd17d-workload-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:53:11.271224 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:11.271194 2556 generic.go:356] "Generic (PLEG): container finished" podID="2fa83c7b-db09-4266-8581-7292a12dd17d" containerID="c28c100ed60205e205498377a8e85082909333ca3d1109a5e1a2110f956e0919" exitCode=0 Jul 15 12:53:11.271410 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:11.271246 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" Jul 15 12:53:11.271410 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:11.271282 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" event={"ID":"2fa83c7b-db09-4266-8581-7292a12dd17d","Type":"ContainerDied","Data":"c28c100ed60205e205498377a8e85082909333ca3d1109a5e1a2110f956e0919"} Jul 15 12:53:11.271410 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:11.271337 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp" event={"ID":"2fa83c7b-db09-4266-8581-7292a12dd17d","Type":"ContainerDied","Data":"61e09b973aca7ec3637b6faf749519215b2175656ee9689ab11eb360f91ed5e7"} Jul 15 12:53:11.271410 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:11.271366 2556 scope.go:117] "RemoveContainer" containerID="c28c100ed60205e205498377a8e85082909333ca3d1109a5e1a2110f956e0919" Jul 15 12:53:11.279957 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:11.279937 2556 scope.go:117] "RemoveContainer" containerID="80e7e1a79c10e4aff953ce2fa5099d3ebf0d7915f60548f4978188423bd54782" Jul 15 12:53:11.287957 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:11.287934 2556 scope.go:117] "RemoveContainer" containerID="c28c100ed60205e205498377a8e85082909333ca3d1109a5e1a2110f956e0919" Jul 15 12:53:11.288231 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:53:11.288212 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c28c100ed60205e205498377a8e85082909333ca3d1109a5e1a2110f956e0919\": container with ID starting with c28c100ed60205e205498377a8e85082909333ca3d1109a5e1a2110f956e0919 not found: ID does not exist" containerID="c28c100ed60205e205498377a8e85082909333ca3d1109a5e1a2110f956e0919" Jul 15 12:53:11.288297 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:11.288239 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c28c100ed60205e205498377a8e85082909333ca3d1109a5e1a2110f956e0919"} err="failed to get container status \"c28c100ed60205e205498377a8e85082909333ca3d1109a5e1a2110f956e0919\": rpc error: code = NotFound desc = could not find container \"c28c100ed60205e205498377a8e85082909333ca3d1109a5e1a2110f956e0919\": container with ID starting with c28c100ed60205e205498377a8e85082909333ca3d1109a5e1a2110f956e0919 not found: ID does not exist" Jul 15 12:53:11.288297 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:11.288257 2556 scope.go:117] "RemoveContainer" containerID="80e7e1a79c10e4aff953ce2fa5099d3ebf0d7915f60548f4978188423bd54782" Jul 15 12:53:11.288488 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:53:11.288468 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e7e1a79c10e4aff953ce2fa5099d3ebf0d7915f60548f4978188423bd54782\": container with ID starting with 80e7e1a79c10e4aff953ce2fa5099d3ebf0d7915f60548f4978188423bd54782 not found: ID does not exist" containerID="80e7e1a79c10e4aff953ce2fa5099d3ebf0d7915f60548f4978188423bd54782" Jul 15 12:53:11.288545 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:11.288495 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e7e1a79c10e4aff953ce2fa5099d3ebf0d7915f60548f4978188423bd54782"} err="failed to get container status \"80e7e1a79c10e4aff953ce2fa5099d3ebf0d7915f60548f4978188423bd54782\": rpc error: code = NotFound desc = could not find container \"80e7e1a79c10e4aff953ce2fa5099d3ebf0d7915f60548f4978188423bd54782\": container with ID starting with 80e7e1a79c10e4aff953ce2fa5099d3ebf0d7915f60548f4978188423bd54782 not found: ID does not exist" Jul 15 12:53:11.289342 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:11.289319 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp"] Jul 15 12:53:11.292247 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:11.292224 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-20d2f8-openshift-default-6479cccfff-q75sp"] Jul 15 12:53:12.854569 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:12.854535 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fa83c7b-db09-4266-8581-7292a12dd17d" path="/var/lib/kubelet/pods/2fa83c7b-db09-4266-8581-7292a12dd17d/volumes" Jul 15 12:53:16.336536 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:16.336500 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt"] Jul 15 12:53:16.336987 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:16.336817 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" podUID="b344f5af-51f3-4ba6-9ca8-676ad0fa6c26" containerName="maas-api" containerID="cri-o://9e86022411c4098786d680d6fdc44fce238b83fb9ad0b50af7054f76a5a8b42a" gracePeriod=30 Jul 15 12:53:16.366989 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:16.366938 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:16.578683 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:16.578655 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" Jul 15 12:53:16.626886 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:16.626799 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b344f5af-51f3-4ba6-9ca8-676ad0fa6c26-maas-api-tls\") pod \"b344f5af-51f3-4ba6-9ca8-676ad0fa6c26\" (UID: \"b344f5af-51f3-4ba6-9ca8-676ad0fa6c26\") " Jul 15 12:53:16.626886 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:16.626840 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b344f5af-51f3-4ba6-9ca8-676ad0fa6c26-openshift-service-ca\") pod \"b344f5af-51f3-4ba6-9ca8-676ad0fa6c26\" (UID: \"b344f5af-51f3-4ba6-9ca8-676ad0fa6c26\") " Jul 15 12:53:16.627109 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:16.626896 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvj4t\" (UniqueName: \"kubernetes.io/projected/b344f5af-51f3-4ba6-9ca8-676ad0fa6c26-kube-api-access-cvj4t\") pod \"b344f5af-51f3-4ba6-9ca8-676ad0fa6c26\" (UID: \"b344f5af-51f3-4ba6-9ca8-676ad0fa6c26\") " Jul 15 12:53:16.627235 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:16.627204 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b344f5af-51f3-4ba6-9ca8-676ad0fa6c26-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b344f5af-51f3-4ba6-9ca8-676ad0fa6c26" (UID: "b344f5af-51f3-4ba6-9ca8-676ad0fa6c26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:53:16.629153 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:16.629130 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b344f5af-51f3-4ba6-9ca8-676ad0fa6c26-kube-api-access-cvj4t" (OuterVolumeSpecName: "kube-api-access-cvj4t") pod "b344f5af-51f3-4ba6-9ca8-676ad0fa6c26" (UID: "b344f5af-51f3-4ba6-9ca8-676ad0fa6c26"). InnerVolumeSpecName "kube-api-access-cvj4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:53:16.629280 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:16.629260 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b344f5af-51f3-4ba6-9ca8-676ad0fa6c26-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b344f5af-51f3-4ba6-9ca8-676ad0fa6c26" (UID: "b344f5af-51f3-4ba6-9ca8-676ad0fa6c26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:53:16.728292 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:16.728263 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvj4t\" (UniqueName: \"kubernetes.io/projected/b344f5af-51f3-4ba6-9ca8-676ad0fa6c26-kube-api-access-cvj4t\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:53:16.728292 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:16.728291 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b344f5af-51f3-4ba6-9ca8-676ad0fa6c26-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:53:16.728492 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:16.728303 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b344f5af-51f3-4ba6-9ca8-676ad0fa6c26-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 12:53:17.301709 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:17.301617 2556 generic.go:356] "Generic (PLEG): container finished" podID="b344f5af-51f3-4ba6-9ca8-676ad0fa6c26" containerID="9e86022411c4098786d680d6fdc44fce238b83fb9ad0b50af7054f76a5a8b42a" exitCode=0 Jul 15 12:53:17.301709 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:17.301674 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" Jul 15 12:53:17.301956 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:17.301701 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" event={"ID":"b344f5af-51f3-4ba6-9ca8-676ad0fa6c26","Type":"ContainerDied","Data":"9e86022411c4098786d680d6fdc44fce238b83fb9ad0b50af7054f76a5a8b42a"} Jul 15 12:53:17.301956 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:17.301762 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt" event={"ID":"b344f5af-51f3-4ba6-9ca8-676ad0fa6c26","Type":"ContainerDied","Data":"f614bcd9b6025f22f5ae239b85d615471ec9b14edd26a4c8104f2cd6df80a95c"} Jul 15 12:53:17.301956 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:17.301817 2556 scope.go:117] "RemoveContainer" containerID="9e86022411c4098786d680d6fdc44fce238b83fb9ad0b50af7054f76a5a8b42a" Jul 15 12:53:17.310319 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:17.310302 2556 scope.go:117] "RemoveContainer" containerID="9e86022411c4098786d680d6fdc44fce238b83fb9ad0b50af7054f76a5a8b42a" Jul 15 12:53:17.310576 ip-10-0-141-45 kubenswrapper[2556]: E0715 12:53:17.310554 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e86022411c4098786d680d6fdc44fce238b83fb9ad0b50af7054f76a5a8b42a\": container with ID starting with 9e86022411c4098786d680d6fdc44fce238b83fb9ad0b50af7054f76a5a8b42a not found: ID does not exist" containerID="9e86022411c4098786d680d6fdc44fce238b83fb9ad0b50af7054f76a5a8b42a" Jul 15 12:53:17.310647 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:17.310582 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e86022411c4098786d680d6fdc44fce238b83fb9ad0b50af7054f76a5a8b42a"} err="failed to get container status \"9e86022411c4098786d680d6fdc44fce238b83fb9ad0b50af7054f76a5a8b42a\": rpc error: code = NotFound desc = could not find container \"9e86022411c4098786d680d6fdc44fce238b83fb9ad0b50af7054f76a5a8b42a\": container with ID starting with 9e86022411c4098786d680d6fdc44fce238b83fb9ad0b50af7054f76a5a8b42a not found: ID does not exist" Jul 15 12:53:17.318095 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:17.318072 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt"] Jul 15 12:53:17.324517 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:17.324493 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-44c6c5-d64c74b94-dpxvt"] Jul 15 12:53:18.854058 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:18.854025 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b344f5af-51f3-4ba6-9ca8-676ad0fa6c26" path="/var/lib/kubelet/pods/b344f5af-51f3-4ba6-9ca8-676ad0fa6c26/volumes" Jul 15 12:53:19.823850 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:19.823804 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:19.924744 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:19.924712 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:22.325236 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:22.325200 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:26.532305 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:26.532253 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:26.652481 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:26.652434 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:31.620927 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:31.620884 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:31.864241 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:31.864205 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:32.216614 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.216579 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:32.509525 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.509490 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp"] Jul 15 12:53:32.510174 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.510150 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b344f5af-51f3-4ba6-9ca8-676ad0fa6c26" containerName="maas-api" Jul 15 12:53:32.510174 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.510174 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b344f5af-51f3-4ba6-9ca8-676ad0fa6c26" containerName="maas-api" Jul 15 12:53:32.510323 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.510194 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a9f6330-6eaf-4bb1-8a93-76b643775026" containerName="maas-api" Jul 15 12:53:32.510323 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.510203 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a9f6330-6eaf-4bb1-8a93-76b643775026" containerName="maas-api" Jul 15 12:53:32.510323 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.510218 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fa83c7b-db09-4266-8581-7292a12dd17d" containerName="istio-proxy" Jul 15 12:53:32.510323 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.510227 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fa83c7b-db09-4266-8581-7292a12dd17d" containerName="istio-proxy" Jul 15 12:53:32.510323 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.510248 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fa83c7b-db09-4266-8581-7292a12dd17d" containerName="istio-proxy" Jul 15 12:53:32.510323 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.510257 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fa83c7b-db09-4266-8581-7292a12dd17d" containerName="istio-proxy" Jul 15 12:53:32.510523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.510350 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fa83c7b-db09-4266-8581-7292a12dd17d" containerName="istio-proxy" Jul 15 12:53:32.510523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.510360 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b344f5af-51f3-4ba6-9ca8-676ad0fa6c26" containerName="maas-api" Jul 15 12:53:32.510523 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.510371 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a9f6330-6eaf-4bb1-8a93-76b643775026" containerName="maas-api" Jul 15 12:53:32.513569 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.513550 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.519213 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.519187 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-e6c75d-openshift-default-dockercfg-5w55p\"" Jul 15 12:53:32.543106 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.543076 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp"] Jul 15 12:53:32.554267 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.554235 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:32.649688 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.649661 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:32.653353 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.653330 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-credential-socket\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.653430 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.653370 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-data\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.653430 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.653397 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-workload-certs\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.653497 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.653453 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzllz\" (UniqueName: \"kubernetes.io/projected/94d67b4e-c227-4f9f-8d74-1e961986f6a6-kube-api-access-mzllz\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.653535 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.653500 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istiod-ca-cert\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.653535 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.653520 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-podinfo\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.653599 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.653538 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-envoy\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.653599 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.653562 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-workload-socket\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.653662 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.653634 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-token\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.754912 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.754863 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-token\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.754912 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.754915 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-credential-socket\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.755166 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.754950 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-data\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.755166 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.754969 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-workload-certs\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.755166 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.754987 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzllz\" (UniqueName: \"kubernetes.io/projected/94d67b4e-c227-4f9f-8d74-1e961986f6a6-kube-api-access-mzllz\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.755166 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.755018 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istiod-ca-cert\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.755166 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.755048 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-podinfo\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.755166 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.755076 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-envoy\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.755166 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.755113 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-workload-socket\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.755487 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.755432 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-data\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.755553 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.755485 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-workload-certs\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.755553 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.755508 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-workload-socket\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.755633 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.755548 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-credential-socket\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.756004 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.755976 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istiod-ca-cert\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.757481 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.757460 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-envoy\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.757648 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.757573 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-podinfo\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.763125 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.763052 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-token\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.763265 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.763248 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzllz\" (UniqueName: \"kubernetes.io/projected/94d67b4e-c227-4f9f-8d74-1e961986f6a6-kube-api-access-mzllz\") pod \"e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.823376 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.823344 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:32.957168 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.957144 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp"] Jul 15 12:53:32.958721 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:53:32.958696 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d67b4e_c227_4f9f_8d74_1e961986f6a6.slice/crio-d80d206b4bdcfb984e811da933fec9fc86ff4e58a68df7747d07cd8d8a0ff499 WatchSource:0}: Error finding container d80d206b4bdcfb984e811da933fec9fc86ff4e58a68df7747d07cd8d8a0ff499: Status 404 returned error can't find the container with id d80d206b4bdcfb984e811da933fec9fc86ff4e58a68df7747d07cd8d8a0ff499 Jul 15 12:53:32.960895 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.960860 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:53:32.961000 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.960924 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:53:32.961000 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:32.960954 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:53:33.374979 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:33.374868 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" event={"ID":"94d67b4e-c227-4f9f-8d74-1e961986f6a6","Type":"ContainerStarted","Data":"b4b54e14477c14c5b372796527b87c12ffb9f93edaecb80a2bccc0bcbf83664e"} Jul 15 12:53:33.374979 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:33.374935 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" event={"ID":"94d67b4e-c227-4f9f-8d74-1e961986f6a6","Type":"ContainerStarted","Data":"d80d206b4bdcfb984e811da933fec9fc86ff4e58a68df7747d07cd8d8a0ff499"} Jul 15 12:53:33.396099 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:33.396036 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" podStartSLOduration=1.396015531 podStartE2EDuration="1.396015531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:53:33.393203344 +0000 UTC m=+2537.027598296" watchObservedRunningTime="2026-07-15 12:53:33.396015531 +0000 UTC m=+2537.030410482" Jul 15 12:53:33.824387 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:33.824352 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:34.828252 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:34.828217 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:35.362661 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.362626 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:35.383333 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.383304 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:35.384331 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.384314 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:35.448504 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.446206 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948"] Jul 15 12:53:35.448504 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.447056 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fa83c7b-db09-4266-8581-7292a12dd17d" containerName="istio-proxy" Jul 15 12:53:35.451715 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.451164 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" Jul 15 12:53:35.453791 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.453751 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bnnf9\"" Jul 15 12:53:35.453791 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.453760 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-e6c75d-serving-cert\"" Jul 15 12:53:35.453996 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.453760 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 12:53:35.454045 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.454035 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 12:53:35.457280 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.457258 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948"] Jul 15 12:53:35.462197 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.462174 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:35.524118 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.524086 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:35.579074 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.579037 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71be30b-f4fa-442c-b0a9-ac8716b77d87-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e6c75d-7775667d57-b8948\" (UID: \"c71be30b-f4fa-442c-b0a9-ac8716b77d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" Jul 15 12:53:35.579074 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.579077 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2f6r\" (UniqueName: \"kubernetes.io/projected/c71be30b-f4fa-442c-b0a9-ac8716b77d87-kube-api-access-j2f6r\") pod \"maas-api-e2e-shared-a-e6c75d-7775667d57-b8948\" (UID: \"c71be30b-f4fa-442c-b0a9-ac8716b77d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" Jul 15 12:53:35.579298 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.579161 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c71be30b-f4fa-442c-b0a9-ac8716b77d87-maas-api-tls\") pod \"maas-api-e2e-shared-a-e6c75d-7775667d57-b8948\" (UID: \"c71be30b-f4fa-442c-b0a9-ac8716b77d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" Jul 15 12:53:35.625029 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.624934 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:35.680140 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.680107 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71be30b-f4fa-442c-b0a9-ac8716b77d87-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e6c75d-7775667d57-b8948\" (UID: \"c71be30b-f4fa-442c-b0a9-ac8716b77d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" Jul 15 12:53:35.680140 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.680139 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2f6r\" (UniqueName: \"kubernetes.io/projected/c71be30b-f4fa-442c-b0a9-ac8716b77d87-kube-api-access-j2f6r\") pod \"maas-api-e2e-shared-a-e6c75d-7775667d57-b8948\" (UID: \"c71be30b-f4fa-442c-b0a9-ac8716b77d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" Jul 15 12:53:35.680328 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.680190 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c71be30b-f4fa-442c-b0a9-ac8716b77d87-maas-api-tls\") pod \"maas-api-e2e-shared-a-e6c75d-7775667d57-b8948\" (UID: \"c71be30b-f4fa-442c-b0a9-ac8716b77d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" Jul 15 12:53:35.680856 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.680834 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71be30b-f4fa-442c-b0a9-ac8716b77d87-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e6c75d-7775667d57-b8948\" (UID: \"c71be30b-f4fa-442c-b0a9-ac8716b77d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" Jul 15 12:53:35.682711 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.682690 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c71be30b-f4fa-442c-b0a9-ac8716b77d87-maas-api-tls\") pod \"maas-api-e2e-shared-a-e6c75d-7775667d57-b8948\" (UID: \"c71be30b-f4fa-442c-b0a9-ac8716b77d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" Jul 15 12:53:35.689106 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.689085 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2f6r\" (UniqueName: \"kubernetes.io/projected/c71be30b-f4fa-442c-b0a9-ac8716b77d87-kube-api-access-j2f6r\") pod \"maas-api-e2e-shared-a-e6c75d-7775667d57-b8948\" (UID: \"c71be30b-f4fa-442c-b0a9-ac8716b77d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" Jul 15 12:53:35.765899 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.765864 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" Jul 15 12:53:35.898685 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.898658 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948"] Jul 15 12:53:35.901283 ip-10-0-141-45 kubenswrapper[2556]: W0715 12:53:35.901253 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71be30b_f4fa_442c_b0a9_ac8716b77d87.slice/crio-3cd2e2bfe2b512259ee8061e3a14c9de7502f79770c0e3e029a340786079098a WatchSource:0}: Error finding container 3cd2e2bfe2b512259ee8061e3a14c9de7502f79770c0e3e029a340786079098a: Status 404 returned error can't find the container with id 3cd2e2bfe2b512259ee8061e3a14c9de7502f79770c0e3e029a340786079098a Jul 15 12:53:35.971355 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:35.971318 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:36.113143 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:36.113108 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:36.388346 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:36.388314 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" event={"ID":"c71be30b-f4fa-442c-b0a9-ac8716b77d87","Type":"ContainerStarted","Data":"3cd2e2bfe2b512259ee8061e3a14c9de7502f79770c0e3e029a340786079098a"} Jul 15 12:53:37.393653 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:37.393619 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" event={"ID":"c71be30b-f4fa-442c-b0a9-ac8716b77d87","Type":"ContainerStarted","Data":"a56a635b650747510b0d6c68ac0edf29c8288d6b18c280b38b8ec5377d63eb13"} Jul 15 12:53:37.394130 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:37.393906 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" Jul 15 12:53:37.411379 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:37.411334 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" podStartSLOduration=1.872507562 podStartE2EDuration="2.411319499s" totalImagesPullingTime="538.811937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:53:37.409258352 +0000 UTC m=+2541.043653300" watchObservedRunningTime="2026-07-15 12:53:37.411319499 +0000 UTC m=+2541.045714612" Jul 15 12:53:39.915566 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:39.915534 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:40.272301 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:40.272266 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:40.632804 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:40.632683 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:41.035982 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:41.035945 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:43.411584 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:43.411548 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" Jul 15 12:53:43.653885 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:43.653851 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:43.732617 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:43.732536 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv"] Jul 15 12:53:43.735977 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:43.735954 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" Jul 15 12:53:43.738354 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:43.738325 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:43.738619 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:43.738585 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-dc94b0-serving-cert\"" Jul 15 12:53:43.746186 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:43.746166 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv"] Jul 15 12:53:43.814798 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:43.814155 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:43.847454 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:43.847422 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9q25\" (UniqueName: \"kubernetes.io/projected/75f76878-16b1-42a5-8b30-5ab293b1870b-kube-api-access-l9q25\") pod \"maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv\" (UID: \"75f76878-16b1-42a5-8b30-5ab293b1870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" Jul 15 12:53:43.847615 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:43.847459 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75f76878-16b1-42a5-8b30-5ab293b1870b-maas-api-tls\") pod \"maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv\" (UID: \"75f76878-16b1-42a5-8b30-5ab293b1870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" Jul 15 12:53:43.847615 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:43.847525 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f76878-16b1-42a5-8b30-5ab293b1870b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv\" (UID: \"75f76878-16b1-42a5-8b30-5ab293b1870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" Jul 15 12:53:43.913193 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:43.913115 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:43.948866 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:43.948833 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9q25\" (UniqueName: \"kubernetes.io/projected/75f76878-16b1-42a5-8b30-5ab293b1870b-kube-api-access-l9q25\") pod \"maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv\" (UID: \"75f76878-16b1-42a5-8b30-5ab293b1870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" Jul 15 12:53:43.948866 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:43.948873 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75f76878-16b1-42a5-8b30-5ab293b1870b-maas-api-tls\") pod \"maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv\" (UID: \"75f76878-16b1-42a5-8b30-5ab293b1870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" Jul 15 12:53:43.949071 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:43.948899 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f76878-16b1-42a5-8b30-5ab293b1870b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv\" (UID: \"75f76878-16b1-42a5-8b30-5ab293b1870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" Jul 15 12:53:43.949507 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:43.949489 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f76878-16b1-42a5-8b30-5ab293b1870b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv\" (UID: \"75f76878-16b1-42a5-8b30-5ab293b1870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" Jul 15 12:53:43.951468 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:43.951450 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75f76878-16b1-42a5-8b30-5ab293b1870b-maas-api-tls\") pod \"maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv\" (UID: \"75f76878-16b1-42a5-8b30-5ab293b1870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" Jul 15 12:53:43.959910 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:43.959889 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9q25\" (UniqueName: \"kubernetes.io/projected/75f76878-16b1-42a5-8b30-5ab293b1870b-kube-api-access-l9q25\") pod \"maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv\" (UID: \"75f76878-16b1-42a5-8b30-5ab293b1870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" Jul 15 12:53:44.048712 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:44.048681 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" Jul 15 12:53:44.224133 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:44.223975 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv"] Jul 15 12:53:44.383382 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:44.383304 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:44.429991 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:44.429956 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" event={"ID":"75f76878-16b1-42a5-8b30-5ab293b1870b","Type":"ContainerStarted","Data":"68a77ccfda4dc9bdbbaba6cd3e572b246d9dacf109a865660723d59738f9cd01"} Jul 15 12:53:44.485063 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:44.485019 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:45.435935 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:45.435895 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" event={"ID":"75f76878-16b1-42a5-8b30-5ab293b1870b","Type":"ContainerStarted","Data":"ef22025e73e8cc502c77b4ef0573661e2c21603408f1a3101562855a9b1806c3"} Jul 15 12:53:45.436317 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:45.435974 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" Jul 15 12:53:45.452421 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:45.452373 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" podStartSLOduration=2.036913952 podStartE2EDuration="2.452358422s" totalImagesPullingTime="415.44447ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:53:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:53:45.450990954 +0000 UTC m=+2549.085385903" watchObservedRunningTime="2026-07-15 12:53:45.452358422 +0000 UTC m=+2549.086753386" Jul 15 12:53:48.199353 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:48.199307 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:49.457510 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:49.457481 2556 generic.go:356] "Generic (PLEG): container finished" podID="94d67b4e-c227-4f9f-8d74-1e961986f6a6" containerID="b4b54e14477c14c5b372796527b87c12ffb9f93edaecb80a2bccc0bcbf83664e" exitCode=0 Jul 15 12:53:49.458016 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:49.457526 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" event={"ID":"94d67b4e-c227-4f9f-8d74-1e961986f6a6","Type":"ContainerDied","Data":"b4b54e14477c14c5b372796527b87c12ffb9f93edaecb80a2bccc0bcbf83664e"} Jul 15 12:53:49.458016 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:49.457889 2556 scope.go:117] "RemoveContainer" containerID="b4b54e14477c14c5b372796527b87c12ffb9f93edaecb80a2bccc0bcbf83664e" Jul 15 12:53:49.458858 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:49.458829 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:53:49.458928 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:49.458901 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:53:49.458968 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:49.458935 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:53:50.383266 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:50.383228 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:50.463588 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:50.463552 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" event={"ID":"94d67b4e-c227-4f9f-8d74-1e961986f6a6","Type":"ContainerStarted","Data":"4b6156ed038fcecca4f472dbad00eb6d5a09a26fb71a14f6a0a15d0994769646"} Jul 15 12:53:50.823958 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:50.823918 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:50.825126 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:50.825095 2556 patch_prober.go:28] interesting pod/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 15 12:53:50.825269 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:50.825149 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" podUID="94d67b4e-c227-4f9f-8d74-1e961986f6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 15 12:53:51.446570 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:51.446543 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" Jul 15 12:53:51.823943 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:51.823908 2556 patch_prober.go:28] interesting pod/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 15 12:53:51.824332 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:51.823958 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" podUID="94d67b4e-c227-4f9f-8d74-1e961986f6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 15 12:53:52.824366 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:52.824324 2556 patch_prober.go:28] interesting pod/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 15 12:53:52.824758 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:52.824399 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" podUID="94d67b4e-c227-4f9f-8d74-1e961986f6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 15 12:53:53.824423 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:53.824379 2556 patch_prober.go:28] interesting pod/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 15 12:53:53.824930 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:53.824451 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" podUID="94d67b4e-c227-4f9f-8d74-1e961986f6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 15 12:53:54.828349 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:54.828318 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:54.828724 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:54.828591 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:54.829328 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:54.829311 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 12:53:56.729906 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:56.729875 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:56.826018 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:53:56.825988 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:54:05.304761 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:54:05.304729 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:54:06.390017 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:54:06.389969 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:54:15.552447 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:54:15.552409 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:54:35.169505 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:54:35.169471 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:55:05.694340 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:55:05.694307 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:55:12.465527 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:55:12.465487 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:56:16.903294 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:56:16.903219 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:56:16.910105 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:56:16.910085 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 12:56:47.796061 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:56:47.796023 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:56:48.679980 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:56:48.679951 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:56:54.841716 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:56:54.841649 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:56:55.586830 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:56:55.586800 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:57:31.366982 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:57:31.366949 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:57:31.468734 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:57:31.468694 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:57:39.974768 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:57:39.974681 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:57:48.620643 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:57:48.620606 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:57:48.670994 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:57:48.670964 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:58:20.609912 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:58:20.609873 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:58:23.079522 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:58:23.079482 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:58:33.451886 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:58:33.451846 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:58:35.442269 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:58:35.442238 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:58:40.647122 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:58:40.647085 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:58:40.741489 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:58:40.741457 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:58:51.990966 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:58:51.990888 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:58:57.467365 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:58:57.467338 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:58:57.570355 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:58:57.570322 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:59:06.487592 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:59:06.487557 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:59:06.587879 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:59:06.587836 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:59:12.321011 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:59:12.320932 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:59:56.560859 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:59:56.560825 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 12:59:57.485029 ip-10-0-141-45 kubenswrapper[2556]: I0715 12:59:57.484998 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 13:00:00.128445 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:00.128413 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735340-sjtpg"] Jul 15 13:00:00.131997 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:00.131966 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e6c75d-29735340-h5jpb"] Jul 15 13:00:00.132151 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:00.132102 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735340-sjtpg" Jul 15 13:00:00.134726 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:00.134710 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kv8zr\"" Jul 15 13:00:00.135260 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:00.135244 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e6c75d-29735340-h5jpb" Jul 15 13:00:00.140497 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:00.140472 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735340-sjtpg"] Jul 15 13:00:00.143136 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:00.143117 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e6c75d-29735340-h5jpb"] Jul 15 13:00:00.144498 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:00.144290 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735340-sjtpg" Jul 15 13:00:00.153544 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:00.153525 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e6c75d-29735340-h5jpb" Jul 15 13:00:00.288608 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:00.288570 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735340-sjtpg"] Jul 15 13:00:00.292611 ip-10-0-141-45 kubenswrapper[2556]: W0715 13:00:00.292547 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd2f326_b0c2_4ff3_acf4_bfab717c1dfa.slice/crio-7c2bdeead5a0970040e9e3878970ac252bdc9b06cb82bc8c7f67b7ce54e8a8d7 WatchSource:0}: Error finding container 7c2bdeead5a0970040e9e3878970ac252bdc9b06cb82bc8c7f67b7ce54e8a8d7: Status 404 returned error can't find the container with id 7c2bdeead5a0970040e9e3878970ac252bdc9b06cb82bc8c7f67b7ce54e8a8d7 Jul 15 13:00:00.319869 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:00.319846 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e6c75d-29735340-h5jpb"] Jul 15 13:00:00.332870 ip-10-0-141-45 kubenswrapper[2556]: W0715 13:00:00.332838 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0edcb415_3ccc_4cfd_b299_466530471a19.slice/crio-612a3d9048fb0692a9a228e11357f3dce5a7f7fee7336f662e72d608adeee92b WatchSource:0}: Error finding container 612a3d9048fb0692a9a228e11357f3dce5a7f7fee7336f662e72d608adeee92b: Status 404 returned error can't find the container with id 612a3d9048fb0692a9a228e11357f3dce5a7f7fee7336f662e72d608adeee92b Jul 15 13:00:01.101403 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:01.101361 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735340-sjtpg" event={"ID":"7cd2f326-b0c2-4ff3-acf4-bfab717c1dfa","Type":"ContainerStarted","Data":"420b824dac2f1fe2dabd28a422ef17816d7406657c4f3be04f451efde48e6ac1"} Jul 15 13:00:01.101610 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:01.101476 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735340-sjtpg" event={"ID":"7cd2f326-b0c2-4ff3-acf4-bfab717c1dfa","Type":"ContainerStarted","Data":"7c2bdeead5a0970040e9e3878970ac252bdc9b06cb82bc8c7f67b7ce54e8a8d7"} Jul 15 13:00:01.103854 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:01.103822 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e6c75d-29735340-h5jpb" event={"ID":"0edcb415-3ccc-4cfd-b299-466530471a19","Type":"ContainerStarted","Data":"2e23bc748d2709fac9cf0258024775b5824ddbc74eff536826a4b78c0715b463"} Jul 15 13:00:01.103993 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:01.103886 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e6c75d-29735340-h5jpb" event={"ID":"0edcb415-3ccc-4cfd-b299-466530471a19","Type":"ContainerStarted","Data":"612a3d9048fb0692a9a228e11357f3dce5a7f7fee7336f662e72d608adeee92b"} Jul 15 13:00:01.116351 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:01.116293 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735340-sjtpg" podStartSLOduration=1.11627835 podStartE2EDuration="1.11627835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:00:01.115902739 +0000 UTC m=+2924.750297688" watchObservedRunningTime="2026-07-15 13:00:01.11627835 +0000 UTC m=+2924.750673298" Jul 15 13:00:01.129872 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:01.129824 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e6c75d-29735340-h5jpb" podStartSLOduration=1.129808768 podStartE2EDuration="1.129808768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:00:01.129628411 +0000 UTC m=+2924.764023359" watchObservedRunningTime="2026-07-15 13:00:01.129808768 +0000 UTC m=+2924.764203717" Jul 15 13:00:02.109788 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:02.109747 2556 generic.go:356] "Generic (PLEG): container finished" podID="0edcb415-3ccc-4cfd-b299-466530471a19" containerID="2e23bc748d2709fac9cf0258024775b5824ddbc74eff536826a4b78c0715b463" exitCode=0 Jul 15 13:00:02.109977 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:02.109839 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e6c75d-29735340-h5jpb" event={"ID":"0edcb415-3ccc-4cfd-b299-466530471a19","Type":"ContainerDied","Data":"2e23bc748d2709fac9cf0258024775b5824ddbc74eff536826a4b78c0715b463"} Jul 15 13:00:02.111282 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:02.111256 2556 generic.go:356] "Generic (PLEG): container finished" podID="7cd2f326-b0c2-4ff3-acf4-bfab717c1dfa" containerID="420b824dac2f1fe2dabd28a422ef17816d7406657c4f3be04f451efde48e6ac1" exitCode=0 Jul 15 13:00:02.111403 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:02.111315 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735340-sjtpg" event={"ID":"7cd2f326-b0c2-4ff3-acf4-bfab717c1dfa","Type":"ContainerDied","Data":"420b824dac2f1fe2dabd28a422ef17816d7406657c4f3be04f451efde48e6ac1"} Jul 15 13:00:03.260016 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:03.259993 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e6c75d-29735340-h5jpb" Jul 15 13:00:03.269973 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:03.269952 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735340-sjtpg" Jul 15 13:00:04.121418 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:04.121392 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735340-sjtpg" Jul 15 13:00:04.121576 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:04.121437 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735340-sjtpg" event={"ID":"7cd2f326-b0c2-4ff3-acf4-bfab717c1dfa","Type":"ContainerDied","Data":"7c2bdeead5a0970040e9e3878970ac252bdc9b06cb82bc8c7f67b7ce54e8a8d7"} Jul 15 13:00:04.121576 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:04.121480 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c2bdeead5a0970040e9e3878970ac252bdc9b06cb82bc8c7f67b7ce54e8a8d7" Jul 15 13:00:04.122724 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:04.122701 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e6c75d-29735340-h5jpb" event={"ID":"0edcb415-3ccc-4cfd-b299-466530471a19","Type":"ContainerDied","Data":"612a3d9048fb0692a9a228e11357f3dce5a7f7fee7336f662e72d608adeee92b"} Jul 15 13:00:04.122838 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:04.122732 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e6c75d-29735340-h5jpb" Jul 15 13:00:04.122838 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:04.122740 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="612a3d9048fb0692a9a228e11357f3dce5a7f7fee7336f662e72d608adeee92b" Jul 15 13:00:18.099591 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:18.099557 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 13:00:18.194761 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:18.194735 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 13:00:27.994723 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:27.994692 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948"] Jul 15 13:00:27.995254 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:27.994896 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" podUID="c71be30b-f4fa-442c-b0a9-ac8716b77d87" containerName="maas-api" containerID="cri-o://a56a635b650747510b0d6c68ac0edf29c8288d6b18c280b38b8ec5377d63eb13" gracePeriod=30 Jul 15 13:00:28.057058 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:28.057026 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 13:00:28.061697 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:28.061670 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e6c75d-29735340-h5jpb"] Jul 15 13:00:28.066131 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:28.066100 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e6c75d-29735340-h5jpb"] Jul 15 13:00:28.237959 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:28.237928 2556 generic.go:356] "Generic (PLEG): container finished" podID="c71be30b-f4fa-442c-b0a9-ac8716b77d87" containerID="a56a635b650747510b0d6c68ac0edf29c8288d6b18c280b38b8ec5377d63eb13" exitCode=0 Jul 15 13:00:28.237959 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:28.237936 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" Jul 15 13:00:28.238158 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:28.237955 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" event={"ID":"c71be30b-f4fa-442c-b0a9-ac8716b77d87","Type":"ContainerDied","Data":"a56a635b650747510b0d6c68ac0edf29c8288d6b18c280b38b8ec5377d63eb13"} Jul 15 13:00:28.238158 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:28.238000 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" event={"ID":"c71be30b-f4fa-442c-b0a9-ac8716b77d87","Type":"ContainerDied","Data":"3cd2e2bfe2b512259ee8061e3a14c9de7502f79770c0e3e029a340786079098a"} Jul 15 13:00:28.238158 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:28.238021 2556 scope.go:117] "RemoveContainer" containerID="a56a635b650747510b0d6c68ac0edf29c8288d6b18c280b38b8ec5377d63eb13" Jul 15 13:00:28.334919 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:28.334888 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2f6r\" (UniqueName: \"kubernetes.io/projected/c71be30b-f4fa-442c-b0a9-ac8716b77d87-kube-api-access-j2f6r\") pod \"c71be30b-f4fa-442c-b0a9-ac8716b77d87\" (UID: \"c71be30b-f4fa-442c-b0a9-ac8716b77d87\") " Jul 15 13:00:28.335105 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:28.334954 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c71be30b-f4fa-442c-b0a9-ac8716b77d87-maas-api-tls\") pod \"c71be30b-f4fa-442c-b0a9-ac8716b77d87\" (UID: \"c71be30b-f4fa-442c-b0a9-ac8716b77d87\") " Jul 15 13:00:28.335105 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:28.334977 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71be30b-f4fa-442c-b0a9-ac8716b77d87-openshift-service-ca\") pod \"c71be30b-f4fa-442c-b0a9-ac8716b77d87\" (UID: \"c71be30b-f4fa-442c-b0a9-ac8716b77d87\") " Jul 15 13:00:28.335400 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:28.335378 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71be30b-f4fa-442c-b0a9-ac8716b77d87-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c71be30b-f4fa-442c-b0a9-ac8716b77d87" (UID: "c71be30b-f4fa-442c-b0a9-ac8716b77d87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:00:28.337157 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:28.337132 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c71be30b-f4fa-442c-b0a9-ac8716b77d87-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c71be30b-f4fa-442c-b0a9-ac8716b77d87" (UID: "c71be30b-f4fa-442c-b0a9-ac8716b77d87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:00:28.337350 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:28.337332 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71be30b-f4fa-442c-b0a9-ac8716b77d87-kube-api-access-j2f6r" (OuterVolumeSpecName: "kube-api-access-j2f6r") pod "c71be30b-f4fa-442c-b0a9-ac8716b77d87" (UID: "c71be30b-f4fa-442c-b0a9-ac8716b77d87"). InnerVolumeSpecName "kube-api-access-j2f6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:00:28.436138 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:28.436106 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71be30b-f4fa-442c-b0a9-ac8716b77d87-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 13:00:28.436138 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:28.436132 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2f6r\" (UniqueName: \"kubernetes.io/projected/c71be30b-f4fa-442c-b0a9-ac8716b77d87-kube-api-access-j2f6r\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 13:00:28.436138 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:28.436142 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c71be30b-f4fa-442c-b0a9-ac8716b77d87-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 13:00:28.853937 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:28.853900 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0edcb415-3ccc-4cfd-b299-466530471a19" path="/var/lib/kubelet/pods/0edcb415-3ccc-4cfd-b299-466530471a19/volumes" Jul 15 13:00:29.242991 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:29.242905 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948" Jul 15 13:00:29.259715 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:29.259689 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948"] Jul 15 13:00:29.265738 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:29.265712 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e6c75d-7775667d57-b8948"] Jul 15 13:00:30.854451 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:30.854419 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c71be30b-f4fa-442c-b0a9-ac8716b77d87" path="/var/lib/kubelet/pods/c71be30b-f4fa-442c-b0a9-ac8716b77d87/volumes" Jul 15 13:00:32.658191 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:32.658151 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 13:00:38.409051 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:38.408944 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 13:00:38.507699 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:38.507662 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 13:00:43.345537 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:43.345503 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp"] Jul 15 13:00:43.345958 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:43.345712 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" podUID="94d67b4e-c227-4f9f-8d74-1e961986f6a6" containerName="istio-proxy" containerID="cri-o://4b6156ed038fcecca4f472dbad00eb6d5a09a26fb71a14f6a0a15d0994769646" gracePeriod=30 Jul 15 13:00:43.369240 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:43.369213 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 13:00:44.953004 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:44.952968 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 13:00:45.053550 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:45.053516 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 13:00:48.602818 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.602768 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 13:00:48.699376 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.699297 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-credential-socket\") pod \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " Jul 15 13:00:48.699376 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.699338 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-data\") pod \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " Jul 15 13:00:48.699605 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.699386 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istiod-ca-cert\") pod \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " Jul 15 13:00:48.699605 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.699428 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-workload-socket\") pod \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " Jul 15 13:00:48.699605 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.699455 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-workload-certs\") pod \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " Jul 15 13:00:48.699605 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.699477 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-envoy\") pod \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " Jul 15 13:00:48.699605 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.699517 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-podinfo\") pod \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " Jul 15 13:00:48.699605 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.699542 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-token\") pod \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " Jul 15 13:00:48.699605 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.699573 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzllz\" (UniqueName: \"kubernetes.io/projected/94d67b4e-c227-4f9f-8d74-1e961986f6a6-kube-api-access-mzllz\") pod \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\" (UID: \"94d67b4e-c227-4f9f-8d74-1e961986f6a6\") " Jul 15 13:00:48.700002 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.699621 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "94d67b4e-c227-4f9f-8d74-1e961986f6a6" (UID: "94d67b4e-c227-4f9f-8d74-1e961986f6a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:00:48.700002 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.699748 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "94d67b4e-c227-4f9f-8d74-1e961986f6a6" (UID: "94d67b4e-c227-4f9f-8d74-1e961986f6a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:00:48.700002 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.699835 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-workload-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 13:00:48.700002 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.699848 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-credential-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 13:00:48.700002 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.699860 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "94d67b4e-c227-4f9f-8d74-1e961986f6a6" (UID: "94d67b4e-c227-4f9f-8d74-1e961986f6a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:00:48.700248 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.700125 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "94d67b4e-c227-4f9f-8d74-1e961986f6a6" (UID: "94d67b4e-c227-4f9f-8d74-1e961986f6a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:00:48.700475 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.700305 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-data" (OuterVolumeSpecName: "istio-data") pod "94d67b4e-c227-4f9f-8d74-1e961986f6a6" (UID: "94d67b4e-c227-4f9f-8d74-1e961986f6a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:00:48.701957 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.701920 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "94d67b4e-c227-4f9f-8d74-1e961986f6a6" (UID: "94d67b4e-c227-4f9f-8d74-1e961986f6a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 13:00:48.702072 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.701956 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-token" (OuterVolumeSpecName: "istio-token") pod "94d67b4e-c227-4f9f-8d74-1e961986f6a6" (UID: "94d67b4e-c227-4f9f-8d74-1e961986f6a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:00:48.702072 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.701960 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d67b4e-c227-4f9f-8d74-1e961986f6a6-kube-api-access-mzllz" (OuterVolumeSpecName: "kube-api-access-mzllz") pod "94d67b4e-c227-4f9f-8d74-1e961986f6a6" (UID: "94d67b4e-c227-4f9f-8d74-1e961986f6a6"). InnerVolumeSpecName "kube-api-access-mzllz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:00:48.702072 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.702003 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "94d67b4e-c227-4f9f-8d74-1e961986f6a6" (UID: "94d67b4e-c227-4f9f-8d74-1e961986f6a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:00:48.800882 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.800857 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istiod-ca-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 13:00:48.800882 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.800879 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-workload-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 13:00:48.800882 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.800887 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-envoy\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 13:00:48.801090 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.800894 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-podinfo\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 13:00:48.801090 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.800901 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 13:00:48.801090 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.800910 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzllz\" (UniqueName: \"kubernetes.io/projected/94d67b4e-c227-4f9f-8d74-1e961986f6a6-kube-api-access-mzllz\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 13:00:48.801090 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:48.800919 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d67b4e-c227-4f9f-8d74-1e961986f6a6-istio-data\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 13:00:49.336335 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:49.336304 2556 generic.go:356] "Generic (PLEG): container finished" podID="94d67b4e-c227-4f9f-8d74-1e961986f6a6" containerID="4b6156ed038fcecca4f472dbad00eb6d5a09a26fb71a14f6a0a15d0994769646" exitCode=0 Jul 15 13:00:49.336516 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:49.336354 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" Jul 15 13:00:49.336516 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:49.336386 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" event={"ID":"94d67b4e-c227-4f9f-8d74-1e961986f6a6","Type":"ContainerDied","Data":"4b6156ed038fcecca4f472dbad00eb6d5a09a26fb71a14f6a0a15d0994769646"} Jul 15 13:00:49.336516 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:49.336438 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp" event={"ID":"94d67b4e-c227-4f9f-8d74-1e961986f6a6","Type":"ContainerDied","Data":"d80d206b4bdcfb984e811da933fec9fc86ff4e58a68df7747d07cd8d8a0ff499"} Jul 15 13:00:49.336516 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:49.336460 2556 scope.go:117] "RemoveContainer" containerID="4b6156ed038fcecca4f472dbad00eb6d5a09a26fb71a14f6a0a15d0994769646" Jul 15 13:00:49.345195 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:49.345174 2556 scope.go:117] "RemoveContainer" containerID="b4b54e14477c14c5b372796527b87c12ffb9f93edaecb80a2bccc0bcbf83664e" Jul 15 13:00:49.353529 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:49.353506 2556 scope.go:117] "RemoveContainer" containerID="4b6156ed038fcecca4f472dbad00eb6d5a09a26fb71a14f6a0a15d0994769646" Jul 15 13:00:49.353853 ip-10-0-141-45 kubenswrapper[2556]: E0715 13:00:49.353804 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b6156ed038fcecca4f472dbad00eb6d5a09a26fb71a14f6a0a15d0994769646\": container with ID starting with 4b6156ed038fcecca4f472dbad00eb6d5a09a26fb71a14f6a0a15d0994769646 not found: ID does not exist" containerID="4b6156ed038fcecca4f472dbad00eb6d5a09a26fb71a14f6a0a15d0994769646" Jul 15 13:00:49.353853 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:49.353836 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b6156ed038fcecca4f472dbad00eb6d5a09a26fb71a14f6a0a15d0994769646"} err="failed to get container status \"4b6156ed038fcecca4f472dbad00eb6d5a09a26fb71a14f6a0a15d0994769646\": rpc error: code = NotFound desc = could not find container \"4b6156ed038fcecca4f472dbad00eb6d5a09a26fb71a14f6a0a15d0994769646\": container with ID starting with 4b6156ed038fcecca4f472dbad00eb6d5a09a26fb71a14f6a0a15d0994769646 not found: ID does not exist" Jul 15 13:00:49.353853 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:49.353856 2556 scope.go:117] "RemoveContainer" containerID="b4b54e14477c14c5b372796527b87c12ffb9f93edaecb80a2bccc0bcbf83664e" Jul 15 13:00:49.354109 ip-10-0-141-45 kubenswrapper[2556]: E0715 13:00:49.354086 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4b54e14477c14c5b372796527b87c12ffb9f93edaecb80a2bccc0bcbf83664e\": container with ID starting with b4b54e14477c14c5b372796527b87c12ffb9f93edaecb80a2bccc0bcbf83664e not found: ID does not exist" containerID="b4b54e14477c14c5b372796527b87c12ffb9f93edaecb80a2bccc0bcbf83664e" Jul 15 13:00:49.354176 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:49.354115 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b54e14477c14c5b372796527b87c12ffb9f93edaecb80a2bccc0bcbf83664e"} err="failed to get container status \"b4b54e14477c14c5b372796527b87c12ffb9f93edaecb80a2bccc0bcbf83664e\": rpc error: code = NotFound desc = could not find container \"b4b54e14477c14c5b372796527b87c12ffb9f93edaecb80a2bccc0bcbf83664e\": container with ID starting with b4b54e14477c14c5b372796527b87c12ffb9f93edaecb80a2bccc0bcbf83664e not found: ID does not exist" Jul 15 13:00:49.355248 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:49.355229 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp"] Jul 15 13:00:49.362156 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:49.362134 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-e6c75d-openshift-default-84d5cc799f-jp8wp"] Jul 15 13:00:50.854282 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:50.854249 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94d67b4e-c227-4f9f-8d74-1e961986f6a6" path="/var/lib/kubelet/pods/94d67b4e-c227-4f9f-8d74-1e961986f6a6/volumes" Jul 15 13:00:54.129547 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:54.128948 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv"] Jul 15 13:00:54.129547 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:54.129173 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" podUID="75f76878-16b1-42a5-8b30-5ab293b1870b" containerName="maas-api" containerID="cri-o://ef22025e73e8cc502c77b4ef0573661e2c21603408f1a3101562855a9b1806c3" gracePeriod=30 Jul 15 13:00:54.162104 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:54.162072 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 13:00:54.366279 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:54.366251 2556 generic.go:356] "Generic (PLEG): container finished" podID="75f76878-16b1-42a5-8b30-5ab293b1870b" containerID="ef22025e73e8cc502c77b4ef0573661e2c21603408f1a3101562855a9b1806c3" exitCode=0 Jul 15 13:00:54.366415 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:54.366325 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" event={"ID":"75f76878-16b1-42a5-8b30-5ab293b1870b","Type":"ContainerDied","Data":"ef22025e73e8cc502c77b4ef0573661e2c21603408f1a3101562855a9b1806c3"} Jul 15 13:00:54.366415 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:54.366378 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" event={"ID":"75f76878-16b1-42a5-8b30-5ab293b1870b","Type":"ContainerDied","Data":"68a77ccfda4dc9bdbbaba6cd3e572b246d9dacf109a865660723d59738f9cd01"} Jul 15 13:00:54.366415 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:54.366403 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68a77ccfda4dc9bdbbaba6cd3e572b246d9dacf109a865660723d59738f9cd01" Jul 15 13:00:54.378444 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:54.378425 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" Jul 15 13:00:54.448508 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:54.448424 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f76878-16b1-42a5-8b30-5ab293b1870b-openshift-service-ca\") pod \"75f76878-16b1-42a5-8b30-5ab293b1870b\" (UID: \"75f76878-16b1-42a5-8b30-5ab293b1870b\") " Jul 15 13:00:54.448508 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:54.448489 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9q25\" (UniqueName: \"kubernetes.io/projected/75f76878-16b1-42a5-8b30-5ab293b1870b-kube-api-access-l9q25\") pod \"75f76878-16b1-42a5-8b30-5ab293b1870b\" (UID: \"75f76878-16b1-42a5-8b30-5ab293b1870b\") " Jul 15 13:00:54.448740 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:54.448557 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75f76878-16b1-42a5-8b30-5ab293b1870b-maas-api-tls\") pod \"75f76878-16b1-42a5-8b30-5ab293b1870b\" (UID: \"75f76878-16b1-42a5-8b30-5ab293b1870b\") " Jul 15 13:00:54.448843 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:54.448749 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f76878-16b1-42a5-8b30-5ab293b1870b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "75f76878-16b1-42a5-8b30-5ab293b1870b" (UID: "75f76878-16b1-42a5-8b30-5ab293b1870b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:00:54.450745 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:54.450717 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f76878-16b1-42a5-8b30-5ab293b1870b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "75f76878-16b1-42a5-8b30-5ab293b1870b" (UID: "75f76878-16b1-42a5-8b30-5ab293b1870b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:00:54.450745 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:54.450719 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f76878-16b1-42a5-8b30-5ab293b1870b-kube-api-access-l9q25" (OuterVolumeSpecName: "kube-api-access-l9q25") pod "75f76878-16b1-42a5-8b30-5ab293b1870b" (UID: "75f76878-16b1-42a5-8b30-5ab293b1870b"). InnerVolumeSpecName "kube-api-access-l9q25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:00:54.549846 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:54.549798 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75f76878-16b1-42a5-8b30-5ab293b1870b-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 13:00:54.549846 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:54.549839 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f76878-16b1-42a5-8b30-5ab293b1870b-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 13:00:54.549846 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:54.549849 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9q25\" (UniqueName: \"kubernetes.io/projected/75f76878-16b1-42a5-8b30-5ab293b1870b-kube-api-access-l9q25\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 15 13:00:55.370117 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:55.370083 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv" Jul 15 13:00:55.387744 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:55.387715 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv"] Jul 15 13:00:55.393801 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:55.393758 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-dc94b0-64d54d887b-hnmnv"] Jul 15 13:00:56.853754 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:56.853720 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75f76878-16b1-42a5-8b30-5ab293b1870b" path="/var/lib/kubelet/pods/75f76878-16b1-42a5-8b30-5ab293b1870b/volumes" Jul 15 13:00:58.841202 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:00:58.841165 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 13:01:04.736640 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:04.736603 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 13:01:04.836638 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:04.836600 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 13:01:09.507812 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:09.507764 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 13:01:09.739350 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:09.739299 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 13:01:15.100976 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:15.100945 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-jwf49_e03cfc96-db62-4148-b246-53dc2077df25/manager/0.log" Jul 15 13:01:15.219758 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:15.219708 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tm7pl_1d92b1bf-91bf-4159-8c51-1f8cb532d7b7/manager/0.log" Jul 15 13:01:15.970725 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:15.970700 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85d97db8bc-zgqt9_34d10c17-5d61-430f-9af9-10289dfd8171/copy-manifests/0.log" Jul 15 13:01:15.978521 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:15.978500 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85d97db8bc-zgqt9_34d10c17-5d61-430f-9af9-10289dfd8171/manager/0.log" Jul 15 13:01:16.688198 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:16.688166 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29735340-sjtpg_7cd2f326-b0c2-4ff3-acf4-bfab717c1dfa/cleanup/0.log" Jul 15 13:01:16.798939 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:16.798915 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-e10a8061-z44x5_9a890c67-9e3d-4c11-b61f-e7ef40b8c68d/revoke-keys/1.log" Jul 15 13:01:16.909045 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:16.909007 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-f7995e41-7hx4k_96b14a39-ac36-4203-853e-c1243e3381d3/revoke-keys/0.log" Jul 15 13:01:16.932858 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:16.932831 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 13:01:16.939906 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:16.939834 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 13:01:17.074319 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:17.074289 2556 scope.go:117] "RemoveContainer" containerID="ef22025e73e8cc502c77b4ef0573661e2c21603408f1a3101562855a9b1806c3" Jul 15 13:01:20.063065 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:20.063030 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-ktxhx_a3fe1727-a76d-4436-9b2c-6dab569e8e26/manager/0.log" Jul 15 13:01:20.316437 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:20.316326 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-vqmb9_681b92ea-b948-41b1-ab29-e6e2044b5cdf/manager/0.log" Jul 15 13:01:20.807708 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:20.807680 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-blmkq_9ed8cc67-c7ae-4c2e-8b81-df7808745d1a/discovery/0.log" Jul 15 13:01:20.923254 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:20.923228 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6795b9c646-8tt8d_7cf9010d-bdc5-4a92-8671-cb5b3e7497f2/kube-auth-proxy/0.log" Jul 15 13:01:21.972922 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:21.972890 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-h79x6_9e7ea4ff-100c-420e-ad9e-ad1327152bd1/main/0.log" Jul 15 13:01:22.321552 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:22.321522 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-xgvnh_5ea3b2c7-baed-4588-8f65-e23603f2a68b/main/0.log" Jul 15 13:01:26.075753 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:26.075721 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 13:01:26.147629 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:26.147602 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" limit=1024 Jul 15 13:01:28.864473 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:28.864445 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sj4hd_24a77a8a-6ec5-4558-8ce8-96ae7f4c9547/global-pull-secret-syncer/0.log" Jul 15 13:01:28.944806 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:28.944761 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4zpzj_da745f4b-00f5-4b21-91f3-64907b52ebea/konnectivity-agent/0.log" Jul 15 13:01:29.067674 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:29.067647 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-45.ec2.internal_a1c7bc3b4c133f1b0578384be47f8d38/haproxy/0.log" Jul 15 13:01:34.151754 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:34.151724 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-ktxhx_a3fe1727-a76d-4436-9b2c-6dab569e8e26/manager/0.log" Jul 15 13:01:34.259898 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:34.259862 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-vqmb9_681b92ea-b948-41b1-ab29-e6e2044b5cdf/manager/0.log" Jul 15 13:01:36.467789 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:36.467691 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m89t4_e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c/node-exporter/0.log" Jul 15 13:01:36.486930 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:36.486905 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m89t4_e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c/kube-rbac-proxy/0.log" Jul 15 13:01:36.507875 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:36.507851 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m89t4_e8432f3a-b5b7-47b2-a530-1d5c31bc8e1c/init-textfile/0.log" Jul 15 13:01:39.331719 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.331681 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b"] Jul 15 13:01:39.332138 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.332051 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f76878-16b1-42a5-8b30-5ab293b1870b" containerName="maas-api" Jul 15 13:01:39.332138 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.332064 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f76878-16b1-42a5-8b30-5ab293b1870b" containerName="maas-api" Jul 15 13:01:39.332138 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.332077 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd2f326-b0c2-4ff3-acf4-bfab717c1dfa" containerName="cleanup" Jul 15 13:01:39.332138 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.332084 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd2f326-b0c2-4ff3-acf4-bfab717c1dfa" containerName="cleanup" Jul 15 13:01:39.332138 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.332096 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94d67b4e-c227-4f9f-8d74-1e961986f6a6" containerName="istio-proxy" Jul 15 13:01:39.332138 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.332102 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="94d67b4e-c227-4f9f-8d74-1e961986f6a6" containerName="istio-proxy" Jul 15 13:01:39.332138 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.332109 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0edcb415-3ccc-4cfd-b299-466530471a19" containerName="cleanup" Jul 15 13:01:39.332138 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.332113 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0edcb415-3ccc-4cfd-b299-466530471a19" containerName="cleanup" Jul 15 13:01:39.332138 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.332120 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94d67b4e-c227-4f9f-8d74-1e961986f6a6" containerName="istio-proxy" Jul 15 13:01:39.332138 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.332125 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="94d67b4e-c227-4f9f-8d74-1e961986f6a6" containerName="istio-proxy" Jul 15 13:01:39.332138 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.332133 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c71be30b-f4fa-442c-b0a9-ac8716b77d87" containerName="maas-api" Jul 15 13:01:39.332138 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.332138 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c71be30b-f4fa-442c-b0a9-ac8716b77d87" containerName="maas-api" Jul 15 13:01:39.332505 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.332192 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c71be30b-f4fa-442c-b0a9-ac8716b77d87" containerName="maas-api" Jul 15 13:01:39.332505 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.332200 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0edcb415-3ccc-4cfd-b299-466530471a19" containerName="cleanup" Jul 15 13:01:39.332505 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.332206 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="94d67b4e-c227-4f9f-8d74-1e961986f6a6" containerName="istio-proxy" Jul 15 13:01:39.332505 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.332211 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f76878-16b1-42a5-8b30-5ab293b1870b" containerName="maas-api" Jul 15 13:01:39.332505 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.332217 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="94d67b4e-c227-4f9f-8d74-1e961986f6a6" containerName="istio-proxy" Jul 15 13:01:39.332505 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.332222 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cd2f326-b0c2-4ff3-acf4-bfab717c1dfa" containerName="cleanup" Jul 15 13:01:39.335278 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.335258 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" Jul 15 13:01:39.338110 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.338070 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vl8nl\"/\"openshift-service-ca.crt\"" Jul 15 13:01:39.338249 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.338229 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vl8nl\"/\"kube-root-ca.crt\"" Jul 15 13:01:39.339379 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.339361 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vl8nl\"/\"default-dockercfg-fdrbd\"" Jul 15 13:01:39.346107 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.346087 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b"] Jul 15 13:01:39.406060 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.406039 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05eac833-d8f5-4194-9d32-bcb6c02d7156-lib-modules\") pod \"perf-node-gather-daemonset-kxn6b\" (UID: \"05eac833-d8f5-4194-9d32-bcb6c02d7156\") " pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" Jul 15 13:01:39.406174 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.406064 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/05eac833-d8f5-4194-9d32-bcb6c02d7156-podres\") pod \"perf-node-gather-daemonset-kxn6b\" (UID: \"05eac833-d8f5-4194-9d32-bcb6c02d7156\") " pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" Jul 15 13:01:39.406174 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.406108 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27xt7\" (UniqueName: \"kubernetes.io/projected/05eac833-d8f5-4194-9d32-bcb6c02d7156-kube-api-access-27xt7\") pod \"perf-node-gather-daemonset-kxn6b\" (UID: \"05eac833-d8f5-4194-9d32-bcb6c02d7156\") " pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" Jul 15 13:01:39.406250 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.406182 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05eac833-d8f5-4194-9d32-bcb6c02d7156-sys\") pod \"perf-node-gather-daemonset-kxn6b\" (UID: \"05eac833-d8f5-4194-9d32-bcb6c02d7156\") " pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" Jul 15 13:01:39.406250 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.406215 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/05eac833-d8f5-4194-9d32-bcb6c02d7156-proc\") pod \"perf-node-gather-daemonset-kxn6b\" (UID: \"05eac833-d8f5-4194-9d32-bcb6c02d7156\") " pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" Jul 15 13:01:39.506733 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.506705 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27xt7\" (UniqueName: \"kubernetes.io/projected/05eac833-d8f5-4194-9d32-bcb6c02d7156-kube-api-access-27xt7\") pod \"perf-node-gather-daemonset-kxn6b\" (UID: \"05eac833-d8f5-4194-9d32-bcb6c02d7156\") " pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" Jul 15 13:01:39.506913 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.506752 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05eac833-d8f5-4194-9d32-bcb6c02d7156-sys\") pod \"perf-node-gather-daemonset-kxn6b\" (UID: \"05eac833-d8f5-4194-9d32-bcb6c02d7156\") " pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" Jul 15 13:01:39.506913 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.506808 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/05eac833-d8f5-4194-9d32-bcb6c02d7156-proc\") pod \"perf-node-gather-daemonset-kxn6b\" (UID: \"05eac833-d8f5-4194-9d32-bcb6c02d7156\") " pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" Jul 15 13:01:39.506913 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.506837 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05eac833-d8f5-4194-9d32-bcb6c02d7156-lib-modules\") pod \"perf-node-gather-daemonset-kxn6b\" (UID: \"05eac833-d8f5-4194-9d32-bcb6c02d7156\") " pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" Jul 15 13:01:39.506913 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.506854 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/05eac833-d8f5-4194-9d32-bcb6c02d7156-podres\") pod \"perf-node-gather-daemonset-kxn6b\" (UID: \"05eac833-d8f5-4194-9d32-bcb6c02d7156\") " pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" Jul 15 13:01:39.507186 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.506927 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/05eac833-d8f5-4194-9d32-bcb6c02d7156-proc\") pod \"perf-node-gather-daemonset-kxn6b\" (UID: \"05eac833-d8f5-4194-9d32-bcb6c02d7156\") " pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" Jul 15 13:01:39.507186 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.506963 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05eac833-d8f5-4194-9d32-bcb6c02d7156-sys\") pod \"perf-node-gather-daemonset-kxn6b\" (UID: \"05eac833-d8f5-4194-9d32-bcb6c02d7156\") " pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" Jul 15 13:01:39.507186 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.506985 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/05eac833-d8f5-4194-9d32-bcb6c02d7156-podres\") pod \"perf-node-gather-daemonset-kxn6b\" (UID: \"05eac833-d8f5-4194-9d32-bcb6c02d7156\") " pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" Jul 15 13:01:39.507186 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.506984 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05eac833-d8f5-4194-9d32-bcb6c02d7156-lib-modules\") pod \"perf-node-gather-daemonset-kxn6b\" (UID: \"05eac833-d8f5-4194-9d32-bcb6c02d7156\") " pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" Jul 15 13:01:39.514612 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.514590 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27xt7\" (UniqueName: \"kubernetes.io/projected/05eac833-d8f5-4194-9d32-bcb6c02d7156-kube-api-access-27xt7\") pod \"perf-node-gather-daemonset-kxn6b\" (UID: \"05eac833-d8f5-4194-9d32-bcb6c02d7156\") " pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" Jul 15 13:01:39.646050 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.645975 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" Jul 15 13:01:39.771408 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:39.771383 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b"] Jul 15 13:01:39.773521 ip-10-0-141-45 kubenswrapper[2556]: W0715 13:01:39.773487 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod05eac833_d8f5_4194_9d32_bcb6c02d7156.slice/crio-0368f7925379043b664d41b44cf3a888d1ee5207af4b592f617b2d952e1c2de2 WatchSource:0}: Error finding container 0368f7925379043b664d41b44cf3a888d1ee5207af4b592f617b2d952e1c2de2: Status 404 returned error can't find the container with id 0368f7925379043b664d41b44cf3a888d1ee5207af4b592f617b2d952e1c2de2 Jul 15 13:01:40.567299 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:40.567260 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" event={"ID":"05eac833-d8f5-4194-9d32-bcb6c02d7156","Type":"ContainerStarted","Data":"58f443f817866b55f460c694ca9b511803177118ed363f51b42a00d6836b99ad"} Jul 15 13:01:40.567826 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:40.567319 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" event={"ID":"05eac833-d8f5-4194-9d32-bcb6c02d7156","Type":"ContainerStarted","Data":"0368f7925379043b664d41b44cf3a888d1ee5207af4b592f617b2d952e1c2de2"} Jul 15 13:01:40.567826 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:40.567384 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" Jul 15 13:01:40.589450 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:40.589403 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" podStartSLOduration=1.5893889190000001 podStartE2EDuration="1.589388919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:01:40.588092346 +0000 UTC m=+3024.222487293" watchObservedRunningTime="2026-07-15 13:01:40.589388919 +0000 UTC m=+3024.223783866" Jul 15 13:01:41.188254 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:41.188215 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b89tf_7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9/dns/0.log" Jul 15 13:01:41.220962 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:41.220929 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b89tf_7478a0e7-eeb9-4da2-8a4b-c7f5cde188b9/kube-rbac-proxy/0.log" Jul 15 13:01:41.409498 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:41.409468 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b8nd7_a15dd1aa-a5bc-4f58-9444-8b1f69f80f5e/dns-node-resolver/0.log" Jul 15 13:01:42.066375 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:42.066340 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p8nqm_6d26d8b5-ec38-45e2-84bb-c2043c7fd9f8/node-ca/0.log" Jul 15 13:01:43.066912 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:43.066881 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-blmkq_9ed8cc67-c7ae-4c2e-8b81-df7808745d1a/discovery/0.log" Jul 15 13:01:43.088985 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:43.088939 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6795b9c646-8tt8d_7cf9010d-bdc5-4a92-8671-cb5b3e7497f2/kube-auth-proxy/0.log" Jul 15 13:01:43.941263 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:43.941238 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fb9dn_989c12ef-5656-4c13-be8d-84a41e5ab36b/serve-healthcheck-canary/0.log" Jul 15 13:01:44.489917 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:44.489874 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nlxxv_cceacc62-64ef-49d1-baac-a24bab3c479c/kube-rbac-proxy/0.log" Jul 15 13:01:44.508920 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:44.508892 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nlxxv_cceacc62-64ef-49d1-baac-a24bab3c479c/exporter/0.log" Jul 15 13:01:44.528178 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:44.528153 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nlxxv_cceacc62-64ef-49d1-baac-a24bab3c479c/extractor/0.log" Jul 15 13:01:46.582259 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:46.582235 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vl8nl/perf-node-gather-daemonset-kxn6b" Jul 15 13:01:46.642397 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:46.642368 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-jwf49_e03cfc96-db62-4148-b246-53dc2077df25/manager/0.log" Jul 15 13:01:46.765268 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:46.765236 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tm7pl_1d92b1bf-91bf-4159-8c51-1f8cb532d7b7/manager/0.log" Jul 15 13:01:47.237531 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:47.237511 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85d97db8bc-zgqt9_34d10c17-5d61-430f-9af9-10289dfd8171/manager/0.log" Jul 15 13:01:47.256131 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:47.256109 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85d97db8bc-zgqt9_34d10c17-5d61-430f-9af9-10289dfd8171/copy-manifests/0.log" Jul 15 13:01:49.848187 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:49.848154 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-blmkq_9ed8cc67-c7ae-4c2e-8b81-df7808745d1a/discovery/0.log" Jul 15 13:01:49.870309 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:49.870287 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6795b9c646-8tt8d_7cf9010d-bdc5-4a92-8671-cb5b3e7497f2/kube-auth-proxy/0.log" Jul 15 13:01:51.364118 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:51.364078 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-blmkq_9ed8cc67-c7ae-4c2e-8b81-df7808745d1a/discovery/0.log" Jul 15 13:01:51.375033 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:51.375008 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6795b9c646-8tt8d_7cf9010d-bdc5-4a92-8671-cb5b3e7497f2/kube-auth-proxy/0.log" Jul 15 13:01:55.143396 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:55.143361 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8ngfj_6132cce3-5328-44d3-af00-a59e66e03f62/kube-multus/0.log" Jul 15 13:01:55.315051 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:55.315027 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k2z7d_757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8/kube-multus-additional-cni-plugins/0.log" Jul 15 13:01:55.334853 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:55.334828 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k2z7d_757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8/egress-router-binary-copy/0.log" Jul 15 13:01:55.356043 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:55.356021 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k2z7d_757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8/cni-plugins/0.log" Jul 15 13:01:55.374066 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:55.374046 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k2z7d_757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8/bond-cni-plugin/0.log" Jul 15 13:01:55.394075 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:55.394003 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k2z7d_757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8/routeoverride-cni/0.log" Jul 15 13:01:55.414269 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:55.414243 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k2z7d_757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8/whereabouts-cni-bincopy/0.log" Jul 15 13:01:55.433045 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:55.433019 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k2z7d_757d71c2-a7e0-4ca7-815a-fa1c18f4a8f8/whereabouts-cni/0.log" Jul 15 13:01:55.661926 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:55.661853 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kfbnz_e79deb5e-778c-420d-a65d-671a7d3d28db/network-metrics-daemon/0.log" Jul 15 13:01:55.679338 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:55.679312 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kfbnz_e79deb5e-778c-420d-a65d-671a7d3d28db/kube-rbac-proxy/0.log" Jul 15 13:01:56.489157 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:56.489123 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-controller/0.log" Jul 15 13:01:56.507096 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:56.507068 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/0.log" Jul 15 13:01:56.507226 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:56.507161 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovn-acl-logging/1.log" Jul 15 13:01:56.531139 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:56.531102 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/kube-rbac-proxy-node/0.log" Jul 15 13:01:56.550149 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:56.550127 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/kube-rbac-proxy-ovn-metrics/0.log" Jul 15 13:01:56.568908 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:56.568891 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/northd/0.log" Jul 15 13:01:56.588624 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:56.588607 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/nbdb/0.log" Jul 15 13:01:56.608123 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:56.608104 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/sbdb/0.log" Jul 15 13:01:56.663679 ip-10-0-141-45 kubenswrapper[2556]: I0715 13:01:56.663652 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fdr4_a434baae-76d0-466a-b7e5-7e0e1e57d90e/ovnkube-controller/0.log"