Jul 06 19:09:40.028089 ip-10-0-133-201 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 19:09:40.028103 ip-10-0-133-201 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 19:09:40.028113 ip-10-0-133-201 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 19:09:40.028435 ip-10-0-133-201 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 19:09:50.142326 ip-10-0-133-201 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 06 19:09:50.142343 ip-10-0-133-201 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 64198c6244a3472f84d9dd313eff8674 -- Jul 06 19:11:42.980024 ip-10-0-133-201 systemd[1]: Starting Kubernetes Kubelet... Jul 06 19:11:43.443302 ip-10-0-133-201 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:11:43.443302 ip-10-0-133-201 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 19:11:43.443302 ip-10-0-133-201 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:11:43.443302 ip-10-0-133-201 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 19:11:43.443302 ip-10-0-133-201 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:11:43.443302 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.415467 2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 19:11:43.443302 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428014 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:11:43.443302 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428036 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:11:43.443302 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428039 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:11:43.443302 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428042 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:11:43.443302 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428046 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:11:43.443302 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428050 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:11:43.443302 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428053 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:11:43.443302 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428056 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:11:43.444950 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428058 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:11:43.444950 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428061 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:11:43.444950 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428063 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:11:43.444950 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428066 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:11:43.444950 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428069 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:11:43.444950 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428071 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:11:43.444950 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428075 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:11:43.444950 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428078 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:11:43.444950 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428080 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:11:43.444950 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428083 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:11:43.444950 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428086 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:11:43.444950 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428089 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:11:43.444950 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428092 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:11:43.444950 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428095 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:11:43.444950 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428097 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:11:43.444950 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428100 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:11:43.444950 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428102 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:11:43.444950 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428105 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:11:43.444950 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428109 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:11:43.445500 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428111 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:11:43.445500 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428114 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:11:43.445500 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428117 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:11:43.445500 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428119 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:11:43.445500 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428123 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:11:43.445500 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428128 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:11:43.445500 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428131 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:11:43.445500 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428134 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:11:43.445500 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428137 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:11:43.445500 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428140 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:11:43.445500 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428144 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:11:43.445500 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428147 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:11:43.445500 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428151 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:11:43.445500 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428154 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:11:43.445500 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428158 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:11:43.445500 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428162 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:11:43.445500 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428165 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:11:43.445500 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428167 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:11:43.445500 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428170 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:11:43.539845 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428173 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:11:43.539845 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428175 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:11:43.539845 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428179 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:11:43.539845 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428182 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:11:43.539845 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428185 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:11:43.539845 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428187 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:11:43.539845 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428191 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:11:43.539845 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428195 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:11:43.539845 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428198 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:11:43.539845 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428201 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:11:43.539845 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428204 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:11:43.539845 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428207 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:11:43.539845 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428209 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:11:43.539845 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428212 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:11:43.539845 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428215 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:11:43.539845 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428217 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:11:43.539845 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428220 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:11:43.539845 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428222 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:11:43.539845 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428225 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:11:43.539845 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428227 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:11:43.502069 ip-10-0-133-201 systemd[1]: Started Kubernetes Kubelet. Jul 06 19:11:43.540884 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428230 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:11:43.540884 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428232 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:11:43.540884 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428235 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:11:43.540884 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428238 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:11:43.540884 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428240 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:11:43.540884 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428243 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:11:43.540884 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428245 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:11:43.540884 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428248 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:11:43.540884 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428251 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:11:43.540884 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428253 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:11:43.540884 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428256 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:11:43.540884 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428258 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:11:43.540884 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428261 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:11:43.540884 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428264 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:11:43.540884 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428267 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:11:43.540884 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428270 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:11:43.540884 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428273 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:11:43.540884 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428276 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:11:43.540884 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428278 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:11:43.540884 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428281 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:11:43.541882 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428284 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:11:43.541882 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428286 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:11:43.541882 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428289 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:11:43.541882 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428292 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:11:43.541882 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428295 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:11:43.541882 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428297 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:11:43.541882 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428300 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:11:43.541882 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428302 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:11:43.541882 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428305 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:11:43.541882 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428308 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:11:43.541882 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428310 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:11:43.541882 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428313 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:11:43.541882 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.428315 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:11:43.541882 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429099 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:11:43.541882 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429108 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:11:43.541882 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429112 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:11:43.541882 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429114 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:11:43.541882 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429117 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:11:43.541882 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429120 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:11:43.541882 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429123 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:11:43.542749 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429125 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:11:43.542749 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429128 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:11:43.542749 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429131 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:11:43.542749 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429134 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:11:43.542749 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429137 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:11:43.542749 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429140 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:11:43.542749 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429143 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:11:43.542749 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429145 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:11:43.542749 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429148 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:11:43.542749 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429151 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:11:43.542749 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429154 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:11:43.542749 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429156 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:11:43.542749 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429159 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:11:43.542749 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429162 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:11:43.542749 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429164 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:11:43.542749 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429167 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:11:43.542749 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429170 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:11:43.542749 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429172 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:11:43.542749 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429175 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:11:43.542749 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429177 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:11:43.543731 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429180 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:11:43.543731 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429183 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:11:43.543731 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429186 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:11:43.543731 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429189 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:11:43.543731 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429192 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:11:43.543731 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429194 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:11:43.543731 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429199 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:11:43.543731 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429203 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:11:43.543731 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429206 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:11:43.543731 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429210 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:11:43.543731 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429214 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:11:43.543731 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429217 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:11:43.543731 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429220 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:11:43.543731 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429223 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:11:43.543731 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429225 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:11:43.543731 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429229 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:11:43.543731 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429232 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:11:43.543731 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429234 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:11:43.543731 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429237 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:11:43.544469 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429240 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:11:43.544469 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429243 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:11:43.544469 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429246 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:11:43.544469 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429248 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:11:43.544469 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429252 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:11:43.544469 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429255 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:11:43.544469 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429257 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:11:43.544469 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429260 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:11:43.544469 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429263 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:11:43.544469 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429265 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:11:43.544469 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429268 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:11:43.544469 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429271 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:11:43.544469 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429274 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:11:43.544469 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429276 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:11:43.544469 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429279 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:11:43.544469 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429282 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:11:43.544469 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429285 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:11:43.544469 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429288 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:11:43.544469 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429290 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:11:43.544469 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429293 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:11:43.545202 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429295 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:11:43.545202 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429298 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:11:43.545202 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429301 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:11:43.545202 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429305 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:11:43.545202 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429307 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:11:43.545202 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429309 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:11:43.545202 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429312 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:11:43.545202 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429314 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:11:43.545202 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429317 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:11:43.545202 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429319 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:11:43.545202 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429322 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:11:43.545202 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429325 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:11:43.545202 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429329 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:11:43.545202 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429336 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:11:43.545202 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429339 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:11:43.545202 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429342 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:11:43.545202 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429345 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:11:43.545202 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429348 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:11:43.545202 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429350 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:11:43.546188 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429353 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:11:43.546188 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429356 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:11:43.546188 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429358 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:11:43.546188 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429361 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:11:43.546188 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429364 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:11:43.546188 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429366 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:11:43.546188 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429369 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:11:43.546188 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429372 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:11:43.546188 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429374 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:11:43.546188 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429377 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:11:43.546188 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429379 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:11:43.546188 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429382 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:11:43.546188 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429384 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:11:43.546188 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429387 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:11:43.546188 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429843 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:11:43.546188 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429850 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:11:43.546188 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429853 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:11:43.546188 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429856 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:11:43.546188 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429859 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:11:43.546188 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429862 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:11:43.547177 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429865 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:11:43.547177 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429868 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:11:43.547177 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429870 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:11:43.547177 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429873 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:11:43.547177 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429877 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:11:43.547177 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429879 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:11:43.547177 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429885 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:11:43.547177 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429888 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:11:43.547177 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429890 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:11:43.547177 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429893 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:11:43.547177 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429896 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:11:43.547177 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429899 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:11:43.547177 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429902 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:11:43.547177 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429904 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:11:43.547177 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429907 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:11:43.547177 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429911 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:11:43.547177 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429914 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:11:43.547177 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429917 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:11:43.547177 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429920 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:11:43.554854 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429923 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:11:43.554854 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429926 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:11:43.554854 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429928 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:11:43.554854 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429931 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:11:43.554854 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429933 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:11:43.554854 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429936 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:11:43.554854 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429938 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:11:43.554854 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429941 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:11:43.554854 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429944 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:11:43.554854 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429946 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:11:43.554854 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429949 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:11:43.554854 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429951 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:11:43.554854 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429954 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:11:43.554854 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429957 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:11:43.554854 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429959 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:11:43.554854 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429961 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:11:43.554854 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429964 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:11:43.554854 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429968 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:11:43.554854 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429971 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:11:43.555470 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429977 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:11:43.555470 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429980 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:11:43.555470 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429983 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:11:43.555470 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429986 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:11:43.555470 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429988 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:11:43.555470 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429991 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:11:43.555470 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429994 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:11:43.555470 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.429997 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:11:43.555470 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430000 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:11:43.555470 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430003 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:11:43.555470 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430006 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:11:43.555470 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430009 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:11:43.555470 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430011 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:11:43.555470 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430013 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:11:43.555470 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430016 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:11:43.555470 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430019 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:11:43.555470 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430021 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:11:43.555470 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430024 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:11:43.555470 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430026 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:11:43.555470 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430029 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:11:43.556175 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430031 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:11:43.556175 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430034 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:11:43.556175 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430036 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:11:43.556175 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430039 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:11:43.556175 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430041 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:11:43.556175 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430044 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:11:43.556175 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430046 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:11:43.556175 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430049 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:11:43.556175 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430051 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:11:43.556175 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430054 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:11:43.556175 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430056 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:11:43.556175 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430061 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:11:43.556175 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430067 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:11:43.556175 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430070 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:11:43.556175 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430073 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:11:43.556175 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430075 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:11:43.556175 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430078 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:11:43.556175 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430081 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:11:43.556175 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430086 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:11:43.556175 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430088 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430091 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430093 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430096 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430098 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430101 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430103 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430106 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430108 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430110 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430113 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430115 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430118 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430120 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430123 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430125 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430193 2566 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430201 2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430210 2566 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430216 2566 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430223 2566 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 19:11:43.562939 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430227 2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430239 2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430245 2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430248 2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430257 2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430260 2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430264 2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430267 2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430270 2566 flags.go:64] FLAG: --cgroup-root="" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430273 2566 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430276 2566 flags.go:64] FLAG: --client-ca-file="" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430280 2566 flags.go:64] FLAG: --cloud-provider="external" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430283 2566 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430288 2566 flags.go:64] FLAG: --cluster-domain="" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430290 2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430293 2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430297 2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430300 2566 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430303 2566 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430306 2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430310 2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430313 2566 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430316 2566 flags.go:64] FLAG: --contention-profiling="false" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430319 2566 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 19:11:43.563573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430321 2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430324 2566 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430328 2566 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430332 2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430335 2566 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430338 2566 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430341 2566 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430344 2566 flags.go:64] FLAG: --enable-server="true" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430347 2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430352 2566 flags.go:64] FLAG: --event-burst="100" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430355 2566 flags.go:64] FLAG: --event-qps="50" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430358 2566 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430363 2566 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430367 2566 flags.go:64] FLAG: --eviction-hard="" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430371 2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430373 2566 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430376 2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430379 2566 flags.go:64] FLAG: --eviction-soft="" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430382 2566 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430385 2566 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430388 2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430391 2566 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430394 2566 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430397 2566 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430400 2566 flags.go:64] FLAG: --feature-gates="" Jul 06 19:11:43.564286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430404 2566 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430407 2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430410 2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430412 2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430416 2566 flags.go:64] FLAG: --healthz-port="10248" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430418 2566 flags.go:64] FLAG: --help="false" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430421 2566 flags.go:64] FLAG: --hostname-override="ip-10-0-133-201.ec2.internal" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430424 2566 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430427 2566 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430430 2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430433 2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430437 2566 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430440 2566 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430442 2566 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430445 2566 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430448 2566 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430451 2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430454 2566 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430457 2566 flags.go:64] FLAG: --kube-reserved="" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430462 2566 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430465 2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430468 2566 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430471 2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430473 2566 flags.go:64] FLAG: --lock-file="" Jul 06 19:11:43.565073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430476 2566 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430479 2566 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430482 2566 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430488 2566 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430491 2566 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430494 2566 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430497 2566 flags.go:64] FLAG: --logging-format="text" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430500 2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430503 2566 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430506 2566 flags.go:64] FLAG: --manifest-url="" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430508 2566 flags.go:64] FLAG: --manifest-url-header="" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430513 2566 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430516 2566 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430520 2566 flags.go:64] FLAG: --max-pods="110" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430524 2566 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430527 2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430530 2566 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430533 2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430536 2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430539 2566 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430541 2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430549 2566 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430552 2566 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430555 2566 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 19:11:43.565821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430558 2566 flags.go:64] FLAG: --pod-cidr="" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430561 2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430568 2566 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430572 2566 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430577 2566 flags.go:64] FLAG: --pods-per-core="0" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430580 2566 flags.go:64] FLAG: --port="10250" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430583 2566 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430586 2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08307f530127737fb" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430589 2566 flags.go:64] FLAG: --qos-reserved="" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430592 2566 flags.go:64] FLAG: --read-only-port="10255" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430595 2566 flags.go:64] FLAG: --register-node="true" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430598 2566 flags.go:64] FLAG: --register-with-taints="" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430601 2566 flags.go:64] FLAG: --registry-burst="10" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430605 2566 flags.go:64] FLAG: --registry-qps="5" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430607 2566 flags.go:64] FLAG: --reserved-cpus="" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430610 2566 flags.go:64] FLAG: --reserved-memory="" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430614 2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430617 2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430620 2566 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430623 2566 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430625 2566 flags.go:64] FLAG: --runonce="false" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430628 2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430631 2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430634 2566 flags.go:64] FLAG: --seccomp-default="false" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430636 2566 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 19:11:43.566534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430639 2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430642 2566 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430645 2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430648 2566 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430651 2566 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430654 2566 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430656 2566 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430659 2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430662 2566 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430665 2566 flags.go:64] FLAG: --system-cgroups="" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430669 2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430675 2566 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430678 2566 flags.go:64] FLAG: --tls-cert-file="" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430681 2566 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430686 2566 flags.go:64] FLAG: --tls-min-version="" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430689 2566 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430691 2566 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430694 2566 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430697 2566 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430701 2566 flags.go:64] FLAG: --v="2" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430704 2566 flags.go:64] FLAG: --version="false" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430709 2566 flags.go:64] FLAG: --vmodule="" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430713 2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.430716 2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430868 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:11:43.567546 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430872 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:11:43.568320 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430875 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:11:43.568320 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430879 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:11:43.568320 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430883 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:11:43.568320 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430885 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:11:43.568320 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430890 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:11:43.568320 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430894 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:11:43.568320 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430898 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:11:43.568320 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430902 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:11:43.568320 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430905 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:11:43.568320 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430908 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:11:43.568320 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430911 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:11:43.568320 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430913 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:11:43.568320 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430916 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:11:43.568320 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430919 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:11:43.568320 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430922 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:11:43.568320 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430924 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:11:43.568320 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430931 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:11:43.568320 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430935 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:11:43.689654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430938 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:11:43.689654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430941 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:11:43.689654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430944 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:11:43.689654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430946 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:11:43.689654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430949 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:11:43.689654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430952 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:11:43.689654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430954 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:11:43.689654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430957 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:11:43.689654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430960 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:11:43.689654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430962 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:11:43.689654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430966 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:11:43.689654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430968 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:11:43.689654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430971 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:11:43.689654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430974 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:11:43.689654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430977 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:11:43.689654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430979 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:11:43.689654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430982 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:11:43.689654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430984 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:11:43.689654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430987 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:11:43.689654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430989 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:11:43.707309 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430992 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:11:43.707309 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430994 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:11:43.707309 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430997 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:11:43.707309 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.430999 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:11:43.707309 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431002 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:11:43.707309 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431004 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:11:43.707309 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431007 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:11:43.707309 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431010 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:11:43.707309 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431012 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:11:43.707309 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431015 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:11:43.707309 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431019 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:11:43.707309 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431022 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:11:43.707309 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431025 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:11:43.707309 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431027 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:11:43.707309 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431030 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:11:43.707309 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431033 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:11:43.707309 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431035 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:11:43.707309 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431038 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:11:43.707309 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431040 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:11:43.707309 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431043 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:11:43.708186 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431045 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:11:43.708186 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431048 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:11:43.708186 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431050 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:11:43.708186 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431053 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:11:43.708186 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431056 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:11:43.708186 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431058 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:11:43.708186 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431061 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:11:43.708186 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431064 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:11:43.708186 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431066 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:11:43.708186 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431069 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:11:43.708186 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431071 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:11:43.708186 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431074 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:11:43.708186 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431076 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:11:43.708186 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431079 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:11:43.708186 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431082 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:11:43.708186 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431084 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:11:43.708186 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431087 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:11:43.708186 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431089 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:11:43.708186 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431092 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:11:43.708186 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431094 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:11:43.709045 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431097 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:11:43.709045 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431099 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:11:43.709045 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431103 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:11:43.709045 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431107 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:11:43.709045 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431110 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:11:43.709045 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431112 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:11:43.709045 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431114 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:11:43.709045 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431117 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:11:43.709045 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431119 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:11:43.709045 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431122 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:11:43.709045 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431124 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:11:43.709045 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431127 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:11:43.709045 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431130 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:11:43.709045 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431132 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:11:43.709045 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431135 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:11:43.709045 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431137 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:11:43.709045 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431140 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:11:43.709045 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431143 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:11:43.709045 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.431145 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:11:43.709700 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.431150 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:11:43.709700 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.438245 2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 19:11:43.709700 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.438262 2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 19:11:43.709700 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.438279 2566 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 19:11:43.709700 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.438285 2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 19:11:43.709700 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438356 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:11:43.709700 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438362 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:11:43.709700 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438366 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:11:43.709700 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438369 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:11:43.709700 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438372 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:11:43.709700 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438374 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:11:43.709700 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438377 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:11:43.709700 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438379 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:11:43.709700 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438382 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:11:43.709700 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438385 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:11:43.710346 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438388 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:11:43.710346 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438391 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:11:43.710346 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438393 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:11:43.710346 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438396 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:11:43.710346 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438398 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:11:43.710346 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438401 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:11:43.710346 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438403 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:11:43.710346 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438406 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:11:43.710346 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438409 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:11:43.710346 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438412 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:11:43.710346 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438415 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:11:43.710346 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438418 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:11:43.710346 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438420 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:11:43.710346 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438423 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:11:43.710346 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438426 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:11:43.710346 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438429 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:11:43.710346 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438432 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:11:43.710346 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438434 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:11:43.710346 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438437 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:11:43.710346 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438439 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:11:43.711039 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438443 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:11:43.711039 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438447 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:11:43.711039 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438451 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:11:43.711039 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438454 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:11:43.711039 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438458 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:11:43.711039 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438461 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:11:43.711039 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438465 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:11:43.711039 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438467 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:11:43.711039 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438470 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:11:43.711039 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438473 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:11:43.711039 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438476 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:11:43.711039 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438479 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:11:43.711039 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438482 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:11:43.711039 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438484 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:11:43.711039 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438487 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:11:43.711039 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438490 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:11:43.711039 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438492 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:11:43.711039 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438495 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:11:43.711039 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438497 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:11:43.711690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438500 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:11:43.711690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438503 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:11:43.711690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438506 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:11:43.711690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438509 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:11:43.711690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438511 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:11:43.711690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438514 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:11:43.711690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438517 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:11:43.711690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438519 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:11:43.711690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438523 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:11:43.711690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438526 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:11:43.711690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438529 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:11:43.711690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438533 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:11:43.711690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438536 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:11:43.711690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438539 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:11:43.711690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438541 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:11:43.711690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438544 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:11:43.711690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438546 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:11:43.711690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438552 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:11:43.711690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438555 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:11:43.711690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438557 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:11:43.712454 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438560 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:11:43.712454 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438562 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:11:43.712454 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438565 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:11:43.712454 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438568 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:11:43.712454 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438571 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:11:43.712454 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438574 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:11:43.712454 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438576 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:11:43.712454 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438579 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:11:43.712454 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438582 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:11:43.712454 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438584 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:11:43.712454 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438587 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:11:43.712454 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438589 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:11:43.712454 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438591 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:11:43.712454 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438596 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:11:43.712454 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438598 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:11:43.712454 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438601 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:11:43.712454 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438604 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:11:43.712454 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438606 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:11:43.712454 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438609 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:11:43.712454 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438611 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:11:43.713136 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438614 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:11:43.713136 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438617 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:11:43.713136 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438620 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:11:43.713136 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438622 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:11:43.713136 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438625 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:11:43.713136 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438627 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:11:43.713136 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438630 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:11:43.713136 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438633 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:11:43.713136 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438636 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:11:43.713136 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438640 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:11:43.713136 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.438645 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:11:43.713136 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438792 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:11:43.713136 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438798 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:11:43.713136 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438802 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:11:43.713136 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438805 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:11:43.713603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438809 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:11:43.713603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438811 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:11:43.713603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438814 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:11:43.713603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438816 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:11:43.713603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438819 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:11:43.713603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438821 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:11:43.713603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438824 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:11:43.713603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438827 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:11:43.713603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438829 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:11:43.713603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438831 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:11:43.713603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438836 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:11:43.713603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438839 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:11:43.713603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438842 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:11:43.713603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438844 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:11:43.713603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438847 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:11:43.713603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438850 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:11:43.713603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438852 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:11:43.713603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438855 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:11:43.713603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438857 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:11:43.713603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438860 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:11:43.714277 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438863 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:11:43.714277 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438865 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:11:43.714277 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438868 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:11:43.714277 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438870 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:11:43.714277 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438873 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:11:43.714277 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438875 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:11:43.714277 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438879 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:11:43.714277 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438883 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:11:43.714277 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438886 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:11:43.714277 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438888 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:11:43.714277 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438891 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:11:43.714277 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438894 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:11:43.714277 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438896 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:11:43.714277 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438900 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:11:43.714277 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438903 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:11:43.714277 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438907 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:11:43.714277 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438909 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:11:43.714277 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438912 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:11:43.714277 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438915 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:11:43.714879 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438918 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:11:43.714879 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438921 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:11:43.714879 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438924 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:11:43.714879 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438928 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:11:43.714879 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438930 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:11:43.714879 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438933 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:11:43.714879 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438935 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:11:43.714879 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438940 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:11:43.714879 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438943 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:11:43.714879 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438946 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:11:43.714879 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438948 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:11:43.714879 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438951 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:11:43.714879 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438953 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:11:43.714879 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438956 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:11:43.714879 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438959 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:11:43.714879 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438962 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:11:43.714879 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438964 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:11:43.714879 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438967 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:11:43.714879 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438969 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:11:43.714879 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438972 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:11:43.715608 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438975 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:11:43.715608 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438978 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:11:43.715608 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438980 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:11:43.715608 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438983 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:11:43.715608 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438986 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:11:43.715608 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438988 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:11:43.715608 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438991 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:11:43.715608 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438993 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:11:43.715608 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438996 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:11:43.715608 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.438999 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:11:43.715608 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439001 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:11:43.715608 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439004 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:11:43.715608 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439006 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:11:43.715608 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439009 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:11:43.715608 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439012 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:11:43.715608 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439017 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:11:43.715608 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439020 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:11:43.715608 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439022 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:11:43.715608 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439025 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:11:43.715608 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439027 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:11:43.716306 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439030 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:11:43.716306 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439033 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:11:43.716306 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439035 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:11:43.716306 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439038 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:11:43.716306 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439040 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:11:43.716306 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439043 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:11:43.716306 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439046 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:11:43.716306 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439049 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:11:43.716306 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439051 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:11:43.716306 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439054 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:11:43.716306 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439057 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:11:43.716306 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439059 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:11:43.716306 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439063 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:11:43.716306 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439065 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:11:43.716306 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439068 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:11:43.716306 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:43.439070 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:11:43.716871 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.439075 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:11:43.716871 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.439186 2566 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 19:11:43.716871 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.441630 2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 19:11:43.716871 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.443104 2566 server.go:1019] "Starting client certificate rotation" Jul 06 19:11:43.716871 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.443211 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:11:43.716871 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.443240 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:11:43.716871 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.446361 2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:11:43.716871 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.447511 2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:11:43.716871 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.467205 2566 log.go:25] "Validated CRI v1 runtime API" Jul 06 19:11:43.716871 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.469957 2566 log.go:25] "Validated CRI v1 image API" Jul 06 19:11:43.716871 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.470760 2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 19:11:43.716871 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.474306 2566 fs.go:135] Filesystem UUIDs: map[0a9077fe-ec43-4112-8feb-b9c68a046331:/dev/nvme0n1p3 45e3db40-5854-437b-a5a6-bba62cff31ee:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 06 19:11:43.717267 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.474335 2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 19:11:43.717389 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.479769 2566 manager.go:216] Machine: {Timestamp:2026-07-06 19:11:43.479006957 +0000 UTC m=+0.374896484 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3118017 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25902f7783eb392e8eaa7d1a6aab3d SystemUUID:ec25902f-7783-eb39-2e8e-aa7d1a6aab3d BootID:64198c62-44a3-472f-84d9-dd313eff8674 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:cb:ef:44:db:b7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:cb:ef:44:db:b7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:73:cd:a3:0f:ba Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 19:11:43.717389 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.479925 2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 19:11:43.717389 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.480071 2566 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 19:11:43.717389 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.483567 2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 19:11:43.717389 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.483594 2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-201.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 19:11:43.717389 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.484374 2566 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 19:11:43.717389 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.484389 2566 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 19:11:43.717389 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.484405 2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.484419 2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.485773 2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.489361 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.489520 2566 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.489561 2566 kubelet.go:486] "Attempting to sync node with API server" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.489595 2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.489614 2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.489593 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.489628 2566 kubelet.go:398] "Adding apiserver pod source" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.489679 2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.490054 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.490070 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.492352 2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.494517 2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.494546 2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.495605 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.495618 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.495624 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.495630 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.495639 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.495644 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.495652 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.495658 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.495665 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.495671 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.495681 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.495686 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.497304 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.497315 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.501110 2566 server.go:1268] "Started kubelet" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.501196 2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.501220 2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.501264 2566 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.506878 2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.508872 2566 server.go:310] "Adding debug handlers to kubelet server" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.511943 2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.512148 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-201.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.512210 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 19:11:43.717689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.512294 2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-201.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.512116 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-201.ec2.internal.18bfc91ceb026f64 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-201.ec2.internal,UID:ip-10-0-133-201.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-201.ec2.internal,},FirstTimestamp:2026-07-06 19:11:43.501078372 +0000 UTC m=+0.396967898,LastTimestamp:2026-07-06 19:11:43.501078372 +0000 UTC m=+0.396967898,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-201.ec2.internal,}" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.514333 2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.514336 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.514453 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-201.ec2.internal\" not found" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.514928 2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.514930 2566 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.514948 2566 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.515040 2566 reconstruct.go:97] "Volume reconstruction finished" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.515046 2566 reconciler.go:29] "Reconciler: start to sync state" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.519856 2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.519867 2566 factory.go:55] Registering systemd factory Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.519874 2566 factory.go:223] Registration of the systemd container factory successfully Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.520119 2566 factory.go:153] Registering CRI-O factory Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.520132 2566 factory.go:223] Registration of the crio container factory successfully Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.520160 2566 factory.go:103] Registering Raw factory Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.520172 2566 manager.go:1196] Started watching for new ooms in manager Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.520567 2566 manager.go:349] Starting recovery of all containers Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.524250 2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-201.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.524274 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.533562 2566 manager.go:354] Recovery completed Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.535080 2566 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.538726 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.541270 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.541304 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.541320 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.541967 2566 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.541980 2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.542002 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.541982 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.542944 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.542972 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:43.718967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.542983 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:43.720311 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.543261 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-201.ec2.internal.18bfc91ced67fe21 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-201.ec2.internal,UID:ip-10-0-133-201.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-201.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-201.ec2.internal,},FirstTimestamp:2026-07-06 19:11:43.541288481 +0000 UTC m=+0.437178007,LastTimestamp:2026-07-06 19:11:43.541288481 +0000 UTC m=+0.437178007,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-201.ec2.internal,}" Jul 06 19:11:43.720311 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.543359 2566 policy_none.go:49] "None policy: Start" Jul 06 19:11:43.720311 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.543377 2566 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 19:11:43.720311 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.543391 2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 19:11:43.720311 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.544468 2566 policy_none.go:47] "Start" Jul 06 19:11:43.720311 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.558098 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-201.ec2.internal.18bfc91ced6856c4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-201.ec2.internal,UID:ip-10-0-133-201.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-201.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-201.ec2.internal,},FirstTimestamp:2026-07-06 19:11:43.541311172 +0000 UTC m=+0.437200923,LastTimestamp:2026-07-06 19:11:43.541311172 +0000 UTC m=+0.437200923,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-201.ec2.internal,}" Jul 06 19:11:43.720311 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.572471 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-crdks" Jul 06 19:11:43.720311 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.575440 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-201.ec2.internal.18bfc91ced688b76 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-201.ec2.internal,UID:ip-10-0-133-201.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-201.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-201.ec2.internal,},FirstTimestamp:2026-07-06 19:11:43.541324662 +0000 UTC m=+0.437214187,LastTimestamp:2026-07-06 19:11:43.541324662 +0000 UTC m=+0.437214187,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-201.ec2.internal,}" Jul 06 19:11:43.720311 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.585763 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-crdks" Jul 06 19:11:43.720311 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.587575 2566 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-201.ec2.internal.18bfc91ced67fe21\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-201.ec2.internal.18bfc91ced67fe21 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-201.ec2.internal,UID:ip-10-0-133-201.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-201.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-201.ec2.internal,},FirstTimestamp:2026-07-06 19:11:43.541288481 +0000 UTC m=+0.437178007,LastTimestamp:2026-07-06 19:11:43.542958256 +0000 UTC m=+0.438847782,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-201.ec2.internal,}" Jul 06 19:11:43.720311 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.605531 2566 manager.go:337] "Starting Device Plugin manager" Jul 06 19:11:43.720311 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.605573 2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 19:11:43.720311 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.605584 2566 server.go:87] "Starting device plugin registration server" Jul 06 19:11:43.720819 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.605861 2566 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 19:11:43.720819 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.605874 2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 19:11:43.720819 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.606047 2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 19:11:43.720819 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.606158 2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 19:11:43.720819 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.606168 2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 19:11:43.720819 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.606605 2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 19:11:43.720819 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.606648 2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-201.ec2.internal\" not found" Jul 06 19:11:43.720819 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.705961 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:43.720819 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.707132 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:43.720819 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.707170 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:43.720819 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.707183 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:43.720819 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.707219 2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-201.ec2.internal" Jul 06 19:11:43.720819 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.715988 2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-201.ec2.internal" Jul 06 19:11:43.720819 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.716014 2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-201.ec2.internal\": node \"ip-10-0-133-201.ec2.internal\" not found" Jul 06 19:11:43.860354 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.734109 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-201.ec2.internal\" not found" Jul 06 19:11:43.860354 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.755154 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 19:11:43.860354 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.756728 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 19:11:43.860354 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.756767 2566 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 19:11:43.860354 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.756789 2566 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 19:11:43.860354 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.756840 2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 19:11:43.860354 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.765296 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:11:43.860354 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.834580 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-201.ec2.internal\" not found" Jul 06 19:11:43.860354 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.858520 2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-201.ec2.internal"] Jul 06 19:11:43.860354 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.858626 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:43.861667 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.860460 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:43.861667 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.860486 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:43.861667 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.860497 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:43.862602 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.861772 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:43.862602 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.861936 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal" Jul 06 19:11:43.862602 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.861974 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:43.866717 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.862673 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:43.866717 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.862691 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:43.866717 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.862700 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:43.866717 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.863059 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:43.866717 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.863081 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:43.866717 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.863096 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:43.866717 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.863926 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-201.ec2.internal" Jul 06 19:11:43.866717 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.863949 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:43.866717 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.864692 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:43.866717 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.864708 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:43.866717 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.864717 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:43.993745 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.913753 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-201.ec2.internal\" not found" node="ip-10-0-133-201.ec2.internal" Jul 06 19:11:43.993745 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.917386 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c744258162850b158aecb328909531ba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal\" (UID: \"c744258162850b158aecb328909531ba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal" Jul 06 19:11:43.993745 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.917409 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c744258162850b158aecb328909531ba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal\" (UID: \"c744258162850b158aecb328909531ba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal" Jul 06 19:11:43.993745 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:43.917428 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a35ae4d1cb32eff9744780fd38da560f-config\") pod \"kube-apiserver-proxy-ip-10-0-133-201.ec2.internal\" (UID: \"a35ae4d1cb32eff9744780fd38da560f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-201.ec2.internal" Jul 06 19:11:43.993745 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.918592 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-201.ec2.internal\" not found" node="ip-10-0-133-201.ec2.internal" Jul 06 19:11:43.993745 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:43.935478 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-201.ec2.internal\" not found" Jul 06 19:11:44.140542 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:44.017507 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c744258162850b158aecb328909531ba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal\" (UID: \"c744258162850b158aecb328909531ba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal" Jul 06 19:11:44.140542 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:44.017533 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a35ae4d1cb32eff9744780fd38da560f-config\") pod \"kube-apiserver-proxy-ip-10-0-133-201.ec2.internal\" (UID: \"a35ae4d1cb32eff9744780fd38da560f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-201.ec2.internal" Jul 06 19:11:44.140542 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:44.017551 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c744258162850b158aecb328909531ba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal\" (UID: \"c744258162850b158aecb328909531ba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal" Jul 06 19:11:44.140542 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:44.017592 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c744258162850b158aecb328909531ba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal\" (UID: \"c744258162850b158aecb328909531ba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal" Jul 06 19:11:44.140542 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:44.017629 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c744258162850b158aecb328909531ba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal\" (UID: \"c744258162850b158aecb328909531ba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal" Jul 06 19:11:44.140542 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:44.017634 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a35ae4d1cb32eff9744780fd38da560f-config\") pod \"kube-apiserver-proxy-ip-10-0-133-201.ec2.internal\" (UID: \"a35ae4d1cb32eff9744780fd38da560f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-201.ec2.internal" Jul 06 19:11:44.140542 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:44.035819 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-201.ec2.internal\" not found" Jul 06 19:11:44.140542 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:44.137096 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-201.ec2.internal\" not found" Jul 06 19:11:44.297392 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:44.220697 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal" Jul 06 19:11:44.297392 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:44.221900 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-201.ec2.internal" Jul 06 19:11:44.297392 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:44.237966 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-201.ec2.internal\" not found" Jul 06 19:11:44.455115 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:44.338799 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-201.ec2.internal\" not found" Jul 06 19:11:44.455115 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:44.440107 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-201.ec2.internal\" not found" Jul 06 19:11:44.455115 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:44.443525 2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 19:11:44.455115 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:44.443690 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:11:44.642342 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:44.489486 2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:11:44.642342 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:44.534300 2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:11:44.642342 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:44.540864 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-201.ec2.internal\" not found" Jul 06 19:11:44.642342 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:44.588046 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 19:06:44 +0000 UTC" deadline="2028-04-16 22:25:56.999176326 +0000 UTC" Jul 06 19:11:44.642342 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:44.588082 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15603h14m12.411097217s" Jul 06 19:11:44.642342 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:44.641656 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-201.ec2.internal\" not found" Jul 06 19:11:44.646659 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:44.642837 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:44.646659 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:44.644400 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:44.646659 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:44.644429 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:44.646659 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:44.644441 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:44.819001 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:44.742137 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-201.ec2.internal\" not found" Jul 06 19:11:44.963171 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:44.843015 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-201.ec2.internal\" not found" Jul 06 19:11:44.963171 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:44.943665 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-201.ec2.internal\" not found" Jul 06 19:11:45.066511 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:44.992862 2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:11:45.066511 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.015470 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-201.ec2.internal" Jul 06 19:11:45.066511 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.027270 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:11:45.066511 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.028336 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal" Jul 06 19:11:45.066511 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.042500 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:11:45.287931 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:45.287898 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda35ae4d1cb32eff9744780fd38da560f.slice/crio-28987f6b85bbd77cee82edb7e90af81af010ab49c7c16be4607b3ff07b3a3575 WatchSource:0}: Error finding container 28987f6b85bbd77cee82edb7e90af81af010ab49c7c16be4607b3ff07b3a3575: Status 404 returned error can't find the container with id 28987f6b85bbd77cee82edb7e90af81af010ab49c7c16be4607b3ff07b3a3575 Jul 06 19:11:45.288121 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:45.288107 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc744258162850b158aecb328909531ba.slice/crio-89f3cff069c13211c52ff52c8727901f1512e5af93d104ba7722e4128092587a WatchSource:0}: Error finding container 89f3cff069c13211c52ff52c8727901f1512e5af93d104ba7722e4128092587a: Status 404 returned error can't find the container with id 89f3cff069c13211c52ff52c8727901f1512e5af93d104ba7722e4128092587a Jul 06 19:11:45.294708 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.294693 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:11:45.492411 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.492318 2566 apiserver.go:52] "Watching apiserver" Jul 06 19:11:45.500082 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.500060 2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 19:11:45.500464 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.500441 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-qrqcb","kube-system/konnectivity-agent-2cnx7","openshift-cluster-node-tuning-operator/tuned-fkb5l","openshift-dns/node-resolver-zcnp6","openshift-image-registry/node-ca-454zs","openshift-multus/multus-additional-cni-plugins-bfxrw","openshift-multus/network-metrics-daemon-2pdkn","openshift-network-diagnostics/network-check-target-zxj86","kube-system/kube-apiserver-proxy-ip-10-0-133-201.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal","openshift-multus/multus-zlp9z","openshift-network-operator/iptables-alerter-snzfv","openshift-ovn-kubernetes/ovnkube-node-2mdkh"] Jul 06 19:11:45.504607 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.504580 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:11:45.504724 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:45.504681 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pdkn" podUID="2db677cb-ce90-46dd-b292-3d8c63da4fbb" Jul 06 19:11:45.506825 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.506802 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-snzfv" Jul 06 19:11:45.509168 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.509145 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.509951 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.509930 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xxs5g\"" Jul 06 19:11:45.510768 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.510751 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:11:45.511343 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.511328 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 19:11:45.511572 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.511553 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zcnp6" Jul 06 19:11:45.513893 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.513872 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-454zs" Jul 06 19:11:45.514607 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.514592 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 19:11:45.515237 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.515212 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 19:11:45.515389 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.515374 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 19:11:45.515531 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.515515 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 19:11:45.516310 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.516291 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.516747 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.516719 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:11:45.516881 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.516746 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 19:11:45.516881 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.516811 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zvgzt\"" Jul 06 19:11:45.516881 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.516725 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fxlq4\"" Jul 06 19:11:45.517054 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.516943 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 19:11:45.517110 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.517059 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 19:11:45.517161 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.517145 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9nznv\"" Jul 06 19:11:45.517832 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.517818 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 19:11:45.518828 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.518808 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.519234 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.519217 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 19:11:45.519357 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.519337 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 19:11:45.519427 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.519365 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 19:11:45.520144 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.520127 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 19:11:45.520226 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.520210 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 19:11:45.521197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.521178 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:11:45.521292 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:45.521261 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrqcb" podUID="4240cab6-4ae1-4d08-ae5d-d3040f4f9d61" Jul 06 19:11:45.522226 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.522199 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 19:11:45.522371 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.522353 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8cb24\"" Jul 06 19:11:45.524202 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.524027 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 19:11:45.524202 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.524167 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 19:11:45.525050 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.524666 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 19:11:45.525050 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.524758 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 19:11:45.525050 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.524834 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 19:11:45.525964 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.525949 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2cnx7" Jul 06 19:11:45.526583 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.526570 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6vfsn\"" Jul 06 19:11:45.526783 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.526761 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f592f36-c320-4ddb-abca-8bc3f6586ae0-serviceca\") pod \"node-ca-454zs\" (UID: \"5f592f36-c320-4ddb-abca-8bc3f6586ae0\") " pod="openshift-image-registry/node-ca-454zs" Jul 06 19:11:45.526870 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.526800 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bkwr\" (UniqueName: \"kubernetes.io/projected/2db677cb-ce90-46dd-b292-3d8c63da4fbb-kube-api-access-2bkwr\") pod \"network-metrics-daemon-2pdkn\" (UID: \"2db677cb-ce90-46dd-b292-3d8c63da4fbb\") " pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:11:45.526870 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.526829 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-modprobe-d\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.526985 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.526885 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5swfc\" (UniqueName: \"kubernetes.io/projected/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-kube-api-access-5swfc\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.526985 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.526922 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a751eaf8-99c8-4e12-92a7-e135a18167a9-tmp-dir\") pod \"node-resolver-zcnp6\" (UID: \"a751eaf8-99c8-4e12-92a7-e135a18167a9\") " pod="openshift-dns/node-resolver-zcnp6" Jul 06 19:11:45.526985 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.526949 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ae8ca85-0e99-47d2-9693-23a51b043fcb-cnibin\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.527094 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.526990 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ae8ca85-0e99-47d2-9693-23a51b043fcb-os-release\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.527094 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527029 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cf4aab1-3038-40fd-9edf-450ab09a1e84-host-slash\") pod \"iptables-alerter-snzfv\" (UID: \"7cf4aab1-3038-40fd-9edf-450ab09a1e84\") " pod="openshift-network-operator/iptables-alerter-snzfv" Jul 06 19:11:45.527094 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527050 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9nrp\" (UniqueName: \"kubernetes.io/projected/7cf4aab1-3038-40fd-9edf-450ab09a1e84-kube-api-access-n9nrp\") pod \"iptables-alerter-snzfv\" (UID: \"7cf4aab1-3038-40fd-9edf-450ab09a1e84\") " pod="openshift-network-operator/iptables-alerter-snzfv" Jul 06 19:11:45.527094 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527068 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-sys\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.527094 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527084 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ae8ca85-0e99-47d2-9693-23a51b043fcb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.527315 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527109 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs\") pod \"network-metrics-daemon-2pdkn\" (UID: \"2db677cb-ce90-46dd-b292-3d8c63da4fbb\") " pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:11:45.527315 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527152 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0ae8ca85-0e99-47d2-9693-23a51b043fcb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.527315 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527196 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-sysconfig\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.527315 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527225 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-sysctl-d\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.527315 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527246 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-var-lib-kubelet\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.527315 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527261 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-host\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.527315 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527276 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-tmp\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.527315 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527290 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a751eaf8-99c8-4e12-92a7-e135a18167a9-hosts-file\") pod \"node-resolver-zcnp6\" (UID: \"a751eaf8-99c8-4e12-92a7-e135a18167a9\") " pod="openshift-dns/node-resolver-zcnp6" Jul 06 19:11:45.527315 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527305 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbzrm\" (UniqueName: \"kubernetes.io/projected/5f592f36-c320-4ddb-abca-8bc3f6586ae0-kube-api-access-fbzrm\") pod \"node-ca-454zs\" (UID: \"5f592f36-c320-4ddb-abca-8bc3f6586ae0\") " pod="openshift-image-registry/node-ca-454zs" Jul 06 19:11:45.527680 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527320 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ae8ca85-0e99-47d2-9693-23a51b043fcb-cni-binary-copy\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.527680 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527337 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0ae8ca85-0e99-47d2-9693-23a51b043fcb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.527680 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527360 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm2jx\" (UniqueName: \"kubernetes.io/projected/0ae8ca85-0e99-47d2-9693-23a51b043fcb-kube-api-access-lm2jx\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.527680 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527391 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7cf4aab1-3038-40fd-9edf-450ab09a1e84-iptables-alerter-script\") pod \"iptables-alerter-snzfv\" (UID: \"7cf4aab1-3038-40fd-9edf-450ab09a1e84\") " pod="openshift-network-operator/iptables-alerter-snzfv" Jul 06 19:11:45.527680 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527410 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-kubernetes\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.527680 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527439 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-lib-modules\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.527680 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527461 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv6ss\" (UniqueName: \"kubernetes.io/projected/a751eaf8-99c8-4e12-92a7-e135a18167a9-kube-api-access-wv6ss\") pod \"node-resolver-zcnp6\" (UID: \"a751eaf8-99c8-4e12-92a7-e135a18167a9\") " pod="openshift-dns/node-resolver-zcnp6" Jul 06 19:11:45.527680 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527485 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-systemd\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.527680 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527519 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-run\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.527680 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527552 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f592f36-c320-4ddb-abca-8bc3f6586ae0-host\") pod \"node-ca-454zs\" (UID: \"5f592f36-c320-4ddb-abca-8bc3f6586ae0\") " pod="openshift-image-registry/node-ca-454zs" Jul 06 19:11:45.527680 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527581 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-sysctl-conf\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.527680 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527599 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-tuned\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.527680 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.527639 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ae8ca85-0e99-47d2-9693-23a51b043fcb-system-cni-dir\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.528624 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.528606 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.530182 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.530167 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 19:11:45.531103 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.531085 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:11:45.531173 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:45.531155 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxj86" podUID="953f279d-10c8-4b4c-97e5-a7bd6ecde126" Jul 06 19:11:45.531325 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.531308 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 19:11:45.531564 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.531551 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 19:11:45.531750 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.531736 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 19:11:45.533568 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.533548 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.533721 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.533691 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cxhq2\"" Jul 06 19:11:45.533818 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.533769 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vt98f\"" Jul 06 19:11:45.533818 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.533792 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 19:11:45.543501 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.543485 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 19:11:45.543808 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.543777 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:11:45.543808 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.543793 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-56pnx\"" Jul 06 19:11:45.575013 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.574991 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tg2f6" Jul 06 19:11:45.589923 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.589902 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tg2f6" Jul 06 19:11:45.616096 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.616074 2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 19:11:45.628218 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628194 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-run-ovn-kubernetes\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.628318 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628224 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-ovn-node-metrics-cert\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.628318 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628243 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-multus-cni-dir\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.628318 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628274 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ae8ca85-0e99-47d2-9693-23a51b043fcb-os-release\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.628318 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628302 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cf4aab1-3038-40fd-9edf-450ab09a1e84-host-slash\") pod \"iptables-alerter-snzfv\" (UID: \"7cf4aab1-3038-40fd-9edf-450ab09a1e84\") " pod="openshift-network-operator/iptables-alerter-snzfv" Jul 06 19:11:45.628476 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628320 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zdq5\" (UniqueName: \"kubernetes.io/projected/e36a6168-595f-4a90-a287-3aa5d4c26d6c-kube-api-access-6zdq5\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.628476 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628335 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-cni-netd\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.628476 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628353 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ae8ca85-0e99-47d2-9693-23a51b043fcb-os-release\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.628476 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628354 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-host-var-lib-cni-bin\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.628476 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628397 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ae8ca85-0e99-47d2-9693-23a51b043fcb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.628476 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628408 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cf4aab1-3038-40fd-9edf-450ab09a1e84-host-slash\") pod \"iptables-alerter-snzfv\" (UID: \"7cf4aab1-3038-40fd-9edf-450ab09a1e84\") " pod="openshift-network-operator/iptables-alerter-snzfv" Jul 06 19:11:45.628476 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628426 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs\") pod \"network-metrics-daemon-2pdkn\" (UID: \"2db677cb-ce90-46dd-b292-3d8c63da4fbb\") " pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:11:45.628726 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628478 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-cnibin\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.628726 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628516 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-multus-socket-dir-parent\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.628726 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:45.628532 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:45.628726 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628550 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-sysconfig\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.628726 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628568 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ae8ca85-0e99-47d2-9693-23a51b043fcb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.628726 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:45.628625 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs podName:2db677cb-ce90-46dd-b292-3d8c63da4fbb nodeName:}" failed. No retries permitted until 2026-07-06 19:11:46.128594849 +0000 UTC m=+3.024484360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs") pod "network-metrics-daemon-2pdkn" (UID: "2db677cb-ce90-46dd-b292-3d8c63da4fbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:45.628726 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628656 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-sysctl-d\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.628726 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628704 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0dba382-5dc7-4c19-bad4-46d9ad717afc-konnectivity-ca\") pod \"konnectivity-agent-2cnx7\" (UID: \"b0dba382-5dc7-4c19-bad4-46d9ad717afc\") " pod="kube-system/konnectivity-agent-2cnx7" Jul 06 19:11:45.629157 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628720 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-sysconfig\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.629157 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628722 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e36a6168-595f-4a90-a287-3aa5d4c26d6c-etc-selinux\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.629157 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628785 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbzrm\" (UniqueName: \"kubernetes.io/projected/5f592f36-c320-4ddb-abca-8bc3f6586ae0-kube-api-access-fbzrm\") pod \"node-ca-454zs\" (UID: \"5f592f36-c320-4ddb-abca-8bc3f6586ae0\") " pod="openshift-image-registry/node-ca-454zs" Jul 06 19:11:45.629157 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628831 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm2jx\" (UniqueName: \"kubernetes.io/projected/0ae8ca85-0e99-47d2-9693-23a51b043fcb-kube-api-access-lm2jx\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.629157 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628854 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7cf4aab1-3038-40fd-9edf-450ab09a1e84-iptables-alerter-script\") pod \"iptables-alerter-snzfv\" (UID: \"7cf4aab1-3038-40fd-9edf-450ab09a1e84\") " pod="openshift-network-operator/iptables-alerter-snzfv" Jul 06 19:11:45.629157 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628873 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-lib-modules\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.629157 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628876 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-sysctl-d\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.629157 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628895 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-run-netns\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.629157 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628916 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-run-systemd\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.629157 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628945 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.629157 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628983 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-lib-modules\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.629157 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.628988 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxfl6\" (UniqueName: \"kubernetes.io/projected/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-kube-api-access-sxfl6\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.629157 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629028 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-systemd\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.629157 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629149 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-etc-openvswitch\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.629730 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629168 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-env-overrides\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.629730 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629198 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5dzh\" (UniqueName: \"kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh\") pod \"network-check-target-zxj86\" (UID: \"953f279d-10c8-4b4c-97e5-a7bd6ecde126\") " pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:11:45.629730 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629222 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-systemd\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.629730 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629264 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-host-run-multus-certs\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.629730 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629289 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-etc-kubernetes\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.629730 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629320 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-sysctl-conf\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.629730 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629357 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-tuned\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.629730 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629438 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-multus-conf-dir\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.629730 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629493 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7cf4aab1-3038-40fd-9edf-450ab09a1e84-iptables-alerter-script\") pod \"iptables-alerter-snzfv\" (UID: \"7cf4aab1-3038-40fd-9edf-450ab09a1e84\") " pod="openshift-network-operator/iptables-alerter-snzfv" Jul 06 19:11:45.629730 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629509 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-sysctl-conf\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.629730 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629559 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ae8ca85-0e99-47d2-9693-23a51b043fcb-system-cni-dir\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.629730 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629610 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ae8ca85-0e99-47d2-9693-23a51b043fcb-system-cni-dir\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.629730 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629649 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e36a6168-595f-4a90-a287-3aa5d4c26d6c-sys-fs\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.629730 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629679 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret\") pod \"global-pull-secret-syncer-qrqcb\" (UID: \"4240cab6-4ae1-4d08-ae5d-d3040f4f9d61\") " pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:11:45.630453 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629723 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f592f36-c320-4ddb-abca-8bc3f6586ae0-serviceca\") pod \"node-ca-454zs\" (UID: \"5f592f36-c320-4ddb-abca-8bc3f6586ae0\") " pod="openshift-image-registry/node-ca-454zs" Jul 06 19:11:45.630453 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629734 2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 19:11:45.630453 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629790 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-modprobe-d\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.630453 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629840 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5swfc\" (UniqueName: \"kubernetes.io/projected/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-kube-api-access-5swfc\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.630453 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629860 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-run-ovn\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.630453 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629912 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-modprobe-d\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.630453 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.629985 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-system-cni-dir\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.630453 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630012 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-os-release\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.630453 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630038 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bec8c587-08a7-467a-8fdc-cfdc44d0571c-cni-binary-copy\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.630453 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630065 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a751eaf8-99c8-4e12-92a7-e135a18167a9-tmp-dir\") pod \"node-resolver-zcnp6\" (UID: \"a751eaf8-99c8-4e12-92a7-e135a18167a9\") " pod="openshift-dns/node-resolver-zcnp6" Jul 06 19:11:45.630453 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630094 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ae8ca85-0e99-47d2-9693-23a51b043fcb-cnibin\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.630453 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630117 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nrp\" (UniqueName: \"kubernetes.io/projected/7cf4aab1-3038-40fd-9edf-450ab09a1e84-kube-api-access-n9nrp\") pod \"iptables-alerter-snzfv\" (UID: \"7cf4aab1-3038-40fd-9edf-450ab09a1e84\") " pod="openshift-network-operator/iptables-alerter-snzfv" Jul 06 19:11:45.630453 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630133 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-sys\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.630453 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630154 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-kubelet-config\") pod \"global-pull-secret-syncer-qrqcb\" (UID: \"4240cab6-4ae1-4d08-ae5d-d3040f4f9d61\") " pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:11:45.630453 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630176 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ae8ca85-0e99-47d2-9693-23a51b043fcb-cnibin\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.631197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630178 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-host-run-k8s-cni-cncf-io\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.631197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630223 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-host-var-lib-cni-multus\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.631197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630243 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-sys\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.631197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630261 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e36a6168-595f-4a90-a287-3aa5d4c26d6c-device-dir\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.631197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630287 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f592f36-c320-4ddb-abca-8bc3f6586ae0-serviceca\") pod \"node-ca-454zs\" (UID: \"5f592f36-c320-4ddb-abca-8bc3f6586ae0\") " pod="openshift-image-registry/node-ca-454zs" Jul 06 19:11:45.631197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630297 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0ae8ca85-0e99-47d2-9693-23a51b043fcb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.631197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630321 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a751eaf8-99c8-4e12-92a7-e135a18167a9-tmp-dir\") pod \"node-resolver-zcnp6\" (UID: \"a751eaf8-99c8-4e12-92a7-e135a18167a9\") " pod="openshift-dns/node-resolver-zcnp6" Jul 06 19:11:45.631197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630324 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-var-lib-kubelet\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.631197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630358 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-host\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.631197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630373 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-var-lib-kubelet\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.631197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630380 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-tmp\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.631197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630403 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-ovnkube-config\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.631197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630419 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-ovnkube-script-lib\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.631197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630440 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-host-var-lib-kubelet\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.631833 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630447 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-host\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.631833 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630467 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bec8c587-08a7-467a-8fdc-cfdc44d0571c-multus-daemon-config\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.631833 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630500 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a751eaf8-99c8-4e12-92a7-e135a18167a9-hosts-file\") pod \"node-resolver-zcnp6\" (UID: \"a751eaf8-99c8-4e12-92a7-e135a18167a9\") " pod="openshift-dns/node-resolver-zcnp6" Jul 06 19:11:45.631833 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630568 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ae8ca85-0e99-47d2-9693-23a51b043fcb-cni-binary-copy\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.631833 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630581 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a751eaf8-99c8-4e12-92a7-e135a18167a9-hosts-file\") pod \"node-resolver-zcnp6\" (UID: \"a751eaf8-99c8-4e12-92a7-e135a18167a9\") " pod="openshift-dns/node-resolver-zcnp6" Jul 06 19:11:45.631833 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630630 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0ae8ca85-0e99-47d2-9693-23a51b043fcb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.631833 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630671 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-kubernetes\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.631833 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630784 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-kubernetes\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.631833 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630854 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-run-openvswitch\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.631833 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630913 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-log-socket\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.631833 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630954 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-cni-bin\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.631833 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630956 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0ae8ca85-0e99-47d2-9693-23a51b043fcb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.631833 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.630991 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-host-run-netns\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.631833 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.631031 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv6ss\" (UniqueName: \"kubernetes.io/projected/a751eaf8-99c8-4e12-92a7-e135a18167a9-kube-api-access-wv6ss\") pod \"node-resolver-zcnp6\" (UID: \"a751eaf8-99c8-4e12-92a7-e135a18167a9\") " pod="openshift-dns/node-resolver-zcnp6" Jul 06 19:11:45.632251 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.631060 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-run\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.632251 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.631112 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e36a6168-595f-4a90-a287-3aa5d4c26d6c-socket-dir\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.632251 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.631144 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e36a6168-595f-4a90-a287-3aa5d4c26d6c-registration-dir\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.632251 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.631167 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-run\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.632251 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.631144 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ae8ca85-0e99-47d2-9693-23a51b043fcb-cni-binary-copy\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.632251 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.631191 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f592f36-c320-4ddb-abca-8bc3f6586ae0-host\") pod \"node-ca-454zs\" (UID: \"5f592f36-c320-4ddb-abca-8bc3f6586ae0\") " pod="openshift-image-registry/node-ca-454zs" Jul 06 19:11:45.632251 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.631206 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0ae8ca85-0e99-47d2-9693-23a51b043fcb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.632251 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.631228 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-dbus\") pod \"global-pull-secret-syncer-qrqcb\" (UID: \"4240cab6-4ae1-4d08-ae5d-d3040f4f9d61\") " pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:11:45.632251 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.631236 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f592f36-c320-4ddb-abca-8bc3f6586ae0-host\") pod \"node-ca-454zs\" (UID: \"5f592f36-c320-4ddb-abca-8bc3f6586ae0\") " pod="openshift-image-registry/node-ca-454zs" Jul 06 19:11:45.632251 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.631266 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td85k\" (UniqueName: \"kubernetes.io/projected/bec8c587-08a7-467a-8fdc-cfdc44d0571c-kube-api-access-td85k\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.632251 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.631302 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e36a6168-595f-4a90-a287-3aa5d4c26d6c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.632251 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.631332 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-kubelet\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.632251 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.631374 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-systemd-units\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.632251 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.631415 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-var-lib-openvswitch\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.632676 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.631443 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-node-log\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.632676 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.631477 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-hostroot\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.632676 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.631507 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bkwr\" (UniqueName: \"kubernetes.io/projected/2db677cb-ce90-46dd-b292-3d8c63da4fbb-kube-api-access-2bkwr\") pod \"network-metrics-daemon-2pdkn\" (UID: \"2db677cb-ce90-46dd-b292-3d8c63da4fbb\") " pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:11:45.632676 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.631550 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0dba382-5dc7-4c19-bad4-46d9ad717afc-agent-certs\") pod \"konnectivity-agent-2cnx7\" (UID: \"b0dba382-5dc7-4c19-bad4-46d9ad717afc\") " pod="kube-system/konnectivity-agent-2cnx7" Jul 06 19:11:45.632676 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.631578 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-slash\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.633987 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.633968 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-etc-tuned\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.634056 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.633987 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-tmp\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.649677 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.649636 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bkwr\" (UniqueName: \"kubernetes.io/projected/2db677cb-ce90-46dd-b292-3d8c63da4fbb-kube-api-access-2bkwr\") pod \"network-metrics-daemon-2pdkn\" (UID: \"2db677cb-ce90-46dd-b292-3d8c63da4fbb\") " pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:11:45.650375 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.650351 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9nrp\" (UniqueName: \"kubernetes.io/projected/7cf4aab1-3038-40fd-9edf-450ab09a1e84-kube-api-access-n9nrp\") pod \"iptables-alerter-snzfv\" (UID: \"7cf4aab1-3038-40fd-9edf-450ab09a1e84\") " pod="openshift-network-operator/iptables-alerter-snzfv" Jul 06 19:11:45.651064 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.651048 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbzrm\" (UniqueName: \"kubernetes.io/projected/5f592f36-c320-4ddb-abca-8bc3f6586ae0-kube-api-access-fbzrm\") pod \"node-ca-454zs\" (UID: \"5f592f36-c320-4ddb-abca-8bc3f6586ae0\") " pod="openshift-image-registry/node-ca-454zs" Jul 06 19:11:45.655526 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.655495 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm2jx\" (UniqueName: \"kubernetes.io/projected/0ae8ca85-0e99-47d2-9693-23a51b043fcb-kube-api-access-lm2jx\") pod \"multus-additional-cni-plugins-bfxrw\" (UID: \"0ae8ca85-0e99-47d2-9693-23a51b043fcb\") " pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.655613 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.655555 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5swfc\" (UniqueName: \"kubernetes.io/projected/02270f78-7b3e-4846-9ee3-9ee6d4cc26fd-kube-api-access-5swfc\") pod \"tuned-fkb5l\" (UID: \"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd\") " pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.655723 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.655703 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv6ss\" (UniqueName: \"kubernetes.io/projected/a751eaf8-99c8-4e12-92a7-e135a18167a9-kube-api-access-wv6ss\") pod \"node-resolver-zcnp6\" (UID: \"a751eaf8-99c8-4e12-92a7-e135a18167a9\") " pod="openshift-dns/node-resolver-zcnp6" Jul 06 19:11:45.732136 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732103 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e36a6168-595f-4a90-a287-3aa5d4c26d6c-sys-fs\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.732330 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732177 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret\") pod \"global-pull-secret-syncer-qrqcb\" (UID: \"4240cab6-4ae1-4d08-ae5d-d3040f4f9d61\") " pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:11:45.732330 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732191 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e36a6168-595f-4a90-a287-3aa5d4c26d6c-sys-fs\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.732330 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732201 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-run-ovn\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.732330 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732231 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-system-cni-dir\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.732330 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732247 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-run-ovn\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.732330 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732248 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-os-release\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.732330 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732291 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-os-release\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.732330 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732303 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bec8c587-08a7-467a-8fdc-cfdc44d0571c-cni-binary-copy\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.732330 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732316 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-system-cni-dir\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.732330 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732334 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-kubelet-config\") pod \"global-pull-secret-syncer-qrqcb\" (UID: \"4240cab6-4ae1-4d08-ae5d-d3040f4f9d61\") " pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:11:45.733100 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732363 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-host-run-k8s-cni-cncf-io\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.733100 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732387 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-host-var-lib-cni-multus\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.733100 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732410 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e36a6168-595f-4a90-a287-3aa5d4c26d6c-device-dir\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.733100 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:45.732417 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:45.733100 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732434 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-host-run-k8s-cni-cncf-io\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.733100 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732435 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-kubelet-config\") pod \"global-pull-secret-syncer-qrqcb\" (UID: \"4240cab6-4ae1-4d08-ae5d-d3040f4f9d61\") " pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:11:45.733100 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732438 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-ovnkube-config\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.733100 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732466 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-host-var-lib-cni-multus\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.733100 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732486 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e36a6168-595f-4a90-a287-3aa5d4c26d6c-device-dir\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.733100 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:45.732491 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret podName:4240cab6-4ae1-4d08-ae5d-d3040f4f9d61 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:46.232467274 +0000 UTC m=+3.128356801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret") pod "global-pull-secret-syncer-qrqcb" (UID: "4240cab6-4ae1-4d08-ae5d-d3040f4f9d61") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:45.733100 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732531 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-ovnkube-script-lib\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.733100 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732549 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-host-var-lib-kubelet\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.733100 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732574 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bec8c587-08a7-467a-8fdc-cfdc44d0571c-multus-daemon-config\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.733100 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732597 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-run-openvswitch\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.733842 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732596 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-host-var-lib-kubelet\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.733842 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732613 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-log-socket\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.733842 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732635 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-cni-bin\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.733842 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732661 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-host-run-netns\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.733842 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732679 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-run-openvswitch\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.733842 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732689 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e36a6168-595f-4a90-a287-3aa5d4c26d6c-socket-dir\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.733842 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732691 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-log-socket\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.733842 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732702 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-cni-bin\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.733842 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732715 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e36a6168-595f-4a90-a287-3aa5d4c26d6c-registration-dir\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.733842 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732744 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-host-run-netns\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.733842 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732743 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-dbus\") pod \"global-pull-secret-syncer-qrqcb\" (UID: \"4240cab6-4ae1-4d08-ae5d-d3040f4f9d61\") " pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:11:45.733842 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732785 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td85k\" (UniqueName: \"kubernetes.io/projected/bec8c587-08a7-467a-8fdc-cfdc44d0571c-kube-api-access-td85k\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.733842 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732801 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e36a6168-595f-4a90-a287-3aa5d4c26d6c-registration-dir\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.733842 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732823 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e36a6168-595f-4a90-a287-3aa5d4c26d6c-socket-dir\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.733842 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732830 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e36a6168-595f-4a90-a287-3aa5d4c26d6c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.734428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732868 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e36a6168-595f-4a90-a287-3aa5d4c26d6c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.734428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732873 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-kubelet\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.734428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732901 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-systemd-units\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.734428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732930 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-var-lib-openvswitch\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.734428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732956 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-node-log\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.734428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732983 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-dbus\") pod \"global-pull-secret-syncer-qrqcb\" (UID: \"4240cab6-4ae1-4d08-ae5d-d3040f4f9d61\") " pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:11:45.734428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732989 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-systemd-units\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.734428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.732989 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bec8c587-08a7-467a-8fdc-cfdc44d0571c-cni-binary-copy\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.734428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733008 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-hostroot\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.734428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733022 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-var-lib-openvswitch\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.734428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733023 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-ovnkube-config\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.734428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733043 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-kubelet\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.734428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733037 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0dba382-5dc7-4c19-bad4-46d9ad717afc-agent-certs\") pod \"konnectivity-agent-2cnx7\" (UID: \"b0dba382-5dc7-4c19-bad4-46d9ad717afc\") " pod="kube-system/konnectivity-agent-2cnx7" Jul 06 19:11:45.734428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733036 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-node-log\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.734428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733046 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-hostroot\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.734911 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733152 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bec8c587-08a7-467a-8fdc-cfdc44d0571c-multus-daemon-config\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.734911 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733157 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-slash\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.734911 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733181 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-slash\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.734911 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733203 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-run-ovn-kubernetes\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.734911 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733226 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-run-ovn-kubernetes\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.734911 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733232 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-ovn-node-metrics-cert\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.734911 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733258 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-multus-cni-dir\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.734911 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733288 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zdq5\" (UniqueName: \"kubernetes.io/projected/e36a6168-595f-4a90-a287-3aa5d4c26d6c-kube-api-access-6zdq5\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.734911 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733314 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-cni-netd\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.734911 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733338 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-host-var-lib-cni-bin\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.734911 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733343 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-multus-cni-dir\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.734911 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733378 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-cnibin\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.734911 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733404 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-multus-socket-dir-parent\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.734911 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733410 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-host-var-lib-cni-bin\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.734911 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733425 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0dba382-5dc7-4c19-bad4-46d9ad717afc-konnectivity-ca\") pod \"konnectivity-agent-2cnx7\" (UID: \"b0dba382-5dc7-4c19-bad4-46d9ad717afc\") " pod="kube-system/konnectivity-agent-2cnx7" Jul 06 19:11:45.735349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733447 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e36a6168-595f-4a90-a287-3aa5d4c26d6c-etc-selinux\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.735349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733466 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-cnibin\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.735349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733475 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-run-netns\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.735349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733499 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-run-systemd\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.735349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733513 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-multus-socket-dir-parent\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.735349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733529 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.735349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733554 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-run-netns\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.735349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733558 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxfl6\" (UniqueName: \"kubernetes.io/projected/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-kube-api-access-sxfl6\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.735349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733381 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-cni-netd\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.735349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733567 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-ovnkube-script-lib\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.735349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733587 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-etc-openvswitch\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.735349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733602 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-run-systemd\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.735349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733638 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.735349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733641 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e36a6168-595f-4a90-a287-3aa5d4c26d6c-etc-selinux\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.735753 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733642 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-etc-openvswitch\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.735753 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733667 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-env-overrides\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.735753 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733697 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5dzh\" (UniqueName: \"kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh\") pod \"network-check-target-zxj86\" (UID: \"953f279d-10c8-4b4c-97e5-a7bd6ecde126\") " pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:11:45.735753 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733727 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-host-run-multus-certs\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.735753 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733752 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-etc-kubernetes\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.735753 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733781 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-multus-conf-dir\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.735753 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733864 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-host-run-multus-certs\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.735753 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733873 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-etc-kubernetes\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.735753 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733919 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bec8c587-08a7-467a-8fdc-cfdc44d0571c-multus-conf-dir\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.735753 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.733995 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0dba382-5dc7-4c19-bad4-46d9ad717afc-konnectivity-ca\") pod \"konnectivity-agent-2cnx7\" (UID: \"b0dba382-5dc7-4c19-bad4-46d9ad717afc\") " pod="kube-system/konnectivity-agent-2cnx7" Jul 06 19:11:45.735753 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.734119 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-env-overrides\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.735753 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.735426 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-ovn-node-metrics-cert\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.735753 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.735593 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0dba382-5dc7-4c19-bad4-46d9ad717afc-agent-certs\") pod \"konnectivity-agent-2cnx7\" (UID: \"b0dba382-5dc7-4c19-bad4-46d9ad717afc\") " pod="kube-system/konnectivity-agent-2cnx7" Jul 06 19:11:45.742662 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:45.742614 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:11:45.742662 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:45.742636 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:11:45.742662 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:45.742651 2566 projected.go:196] Error preparing data for projected volume kube-api-access-c5dzh for pod openshift-network-diagnostics/network-check-target-zxj86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:45.742875 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:45.742720 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh podName:953f279d-10c8-4b4c-97e5-a7bd6ecde126 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:46.242706092 +0000 UTC m=+3.138595608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-c5dzh" (UniqueName: "kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh") pod "network-check-target-zxj86" (UID: "953f279d-10c8-4b4c-97e5-a7bd6ecde126") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:45.744591 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.744569 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td85k\" (UniqueName: \"kubernetes.io/projected/bec8c587-08a7-467a-8fdc-cfdc44d0571c-kube-api-access-td85k\") pod \"multus-zlp9z\" (UID: \"bec8c587-08a7-467a-8fdc-cfdc44d0571c\") " pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.744852 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.744831 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxfl6\" (UniqueName: \"kubernetes.io/projected/a2d615bf-2066-4a10-b984-d7c2be1c8b6d-kube-api-access-sxfl6\") pod \"ovnkube-node-2mdkh\" (UID: \"a2d615bf-2066-4a10-b984-d7c2be1c8b6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.745268 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.745252 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zdq5\" (UniqueName: \"kubernetes.io/projected/e36a6168-595f-4a90-a287-3aa5d4c26d6c-kube-api-access-6zdq5\") pod \"aws-ebs-csi-driver-node-v4l4v\" (UID: \"e36a6168-595f-4a90-a287-3aa5d4c26d6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.761038 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.760998 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal" event={"ID":"c744258162850b158aecb328909531ba","Type":"ContainerStarted","Data":"89f3cff069c13211c52ff52c8727901f1512e5af93d104ba7722e4128092587a"} Jul 06 19:11:45.762171 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.762150 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-201.ec2.internal" event={"ID":"a35ae4d1cb32eff9744780fd38da560f","Type":"ContainerStarted","Data":"28987f6b85bbd77cee82edb7e90af81af010ab49c7c16be4607b3ff07b3a3575"} Jul 06 19:11:45.818013 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.817979 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-snzfv" Jul 06 19:11:45.826894 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.826873 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" Jul 06 19:11:45.828825 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:45.828794 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf4aab1_3038_40fd_9edf_450ab09a1e84.slice/crio-48113c226ccb07b497e7afbec85417f0868b9d489cf4a9902feafc38d615a3ad WatchSource:0}: Error finding container 48113c226ccb07b497e7afbec85417f0868b9d489cf4a9902feafc38d615a3ad: Status 404 returned error can't find the container with id 48113c226ccb07b497e7afbec85417f0868b9d489cf4a9902feafc38d615a3ad Jul 06 19:11:45.833248 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.833227 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zcnp6" Jul 06 19:11:45.835709 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:45.835684 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02270f78_7b3e_4846_9ee3_9ee6d4cc26fd.slice/crio-93bfe8f48aaec381ca27b4ba76b42511ebbaa87c112bb5bddd41d3eade6b7d0d WatchSource:0}: Error finding container 93bfe8f48aaec381ca27b4ba76b42511ebbaa87c112bb5bddd41d3eade6b7d0d: Status 404 returned error can't find the container with id 93bfe8f48aaec381ca27b4ba76b42511ebbaa87c112bb5bddd41d3eade6b7d0d Jul 06 19:11:45.838649 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.838628 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-454zs" Jul 06 19:11:45.843883 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:45.843853 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda751eaf8_99c8_4e12_92a7_e135a18167a9.slice/crio-7f438404aae95d90244c17fa4e8f7750b3957f41e48058f043e0172cd38395aa WatchSource:0}: Error finding container 7f438404aae95d90244c17fa4e8f7750b3957f41e48058f043e0172cd38395aa: Status 404 returned error can't find the container with id 7f438404aae95d90244c17fa4e8f7750b3957f41e48058f043e0172cd38395aa Jul 06 19:11:45.844324 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.844309 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bfxrw" Jul 06 19:11:45.848048 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:45.848025 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f592f36_c320_4ddb_abca_8bc3f6586ae0.slice/crio-baead8bb81bd3899be9df5864ab77ac4cb0a313703721f16f39091c2e8baa9ee WatchSource:0}: Error finding container baead8bb81bd3899be9df5864ab77ac4cb0a313703721f16f39091c2e8baa9ee: Status 404 returned error can't find the container with id baead8bb81bd3899be9df5864ab77ac4cb0a313703721f16f39091c2e8baa9ee Jul 06 19:11:45.850189 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.850169 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:11:45.854101 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:45.854066 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae8ca85_0e99_47d2_9693_23a51b043fcb.slice/crio-7a00a859787293e453face06732acacf18a5008f2631eb120d5ba75bdd065061 WatchSource:0}: Error finding container 7a00a859787293e453face06732acacf18a5008f2631eb120d5ba75bdd065061: Status 404 returned error can't find the container with id 7a00a859787293e453face06732acacf18a5008f2631eb120d5ba75bdd065061 Jul 06 19:11:45.856851 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.856834 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2cnx7" Jul 06 19:11:45.860106 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:45.860067 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d615bf_2066_4a10_b984_d7c2be1c8b6d.slice/crio-092b0430e6e35f47a803d5d9a43031b04308f7894ca2a4dd10ed342c19abb54a WatchSource:0}: Error finding container 092b0430e6e35f47a803d5d9a43031b04308f7894ca2a4dd10ed342c19abb54a: Status 404 returned error can't find the container with id 092b0430e6e35f47a803d5d9a43031b04308f7894ca2a4dd10ed342c19abb54a Jul 06 19:11:45.863482 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.863055 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" Jul 06 19:11:45.867083 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.867064 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zlp9z" Jul 06 19:11:45.868680 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:45.868649 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0dba382_5dc7_4c19_bad4_46d9ad717afc.slice/crio-601b89cbe54fdec68e830c3c43a5d912f13be6bfbff18da4c7dd5389ac3458b3 WatchSource:0}: Error finding container 601b89cbe54fdec68e830c3c43a5d912f13be6bfbff18da4c7dd5389ac3458b3: Status 404 returned error can't find the container with id 601b89cbe54fdec68e830c3c43a5d912f13be6bfbff18da4c7dd5389ac3458b3 Jul 06 19:11:45.879973 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:11:45.879878 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec8c587_08a7_467a_8fdc_cfdc44d0571c.slice/crio-9f70de4188f2fa4240e220ff7ecc781f63a700e0b1faa278dfa1e4fcd53ee273 WatchSource:0}: Error finding container 9f70de4188f2fa4240e220ff7ecc781f63a700e0b1faa278dfa1e4fcd53ee273: Status 404 returned error can't find the container with id 9f70de4188f2fa4240e220ff7ecc781f63a700e0b1faa278dfa1e4fcd53ee273 Jul 06 19:11:45.988142 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:45.987914 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:11:46.136936 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:46.136843 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs\") pod \"network-metrics-daemon-2pdkn\" (UID: \"2db677cb-ce90-46dd-b292-3d8c63da4fbb\") " pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:11:46.137098 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:46.137061 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:46.137175 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:46.137129 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs podName:2db677cb-ce90-46dd-b292-3d8c63da4fbb nodeName:}" failed. No retries permitted until 2026-07-06 19:11:47.137107766 +0000 UTC m=+4.032997299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs") pod "network-metrics-daemon-2pdkn" (UID: "2db677cb-ce90-46dd-b292-3d8c63da4fbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:46.237956 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:46.237916 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret\") pod \"global-pull-secret-syncer-qrqcb\" (UID: \"4240cab6-4ae1-4d08-ae5d-d3040f4f9d61\") " pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:11:46.238150 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:46.238133 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:46.238214 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:46.238203 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret podName:4240cab6-4ae1-4d08-ae5d-d3040f4f9d61 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:47.238184303 +0000 UTC m=+4.134073829 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret") pod "global-pull-secret-syncer-qrqcb" (UID: "4240cab6-4ae1-4d08-ae5d-d3040f4f9d61") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:46.339085 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:46.339045 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5dzh\" (UniqueName: \"kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh\") pod \"network-check-target-zxj86\" (UID: \"953f279d-10c8-4b4c-97e5-a7bd6ecde126\") " pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:11:46.339278 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:46.339246 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:11:46.339278 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:46.339273 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:11:46.339391 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:46.339287 2566 projected.go:196] Error preparing data for projected volume kube-api-access-c5dzh for pod openshift-network-diagnostics/network-check-target-zxj86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:46.339391 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:46.339347 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh podName:953f279d-10c8-4b4c-97e5-a7bd6ecde126 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:47.339328834 +0000 UTC m=+4.235218361 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-c5dzh" (UniqueName: "kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh") pod "network-check-target-zxj86" (UID: "953f279d-10c8-4b4c-97e5-a7bd6ecde126") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:46.593797 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:46.593655 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:06:46 +0000 UTC" deadline="2028-02-24 04:58:20.070112452 +0000 UTC" Jul 06 19:11:46.593797 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:46.593742 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14337h46m33.476374867s" Jul 06 19:11:46.757661 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:46.757620 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:11:46.757879 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:46.757793 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxj86" podUID="953f279d-10c8-4b4c-97e5-a7bd6ecde126" Jul 06 19:11:46.779283 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:46.779203 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bfxrw" event={"ID":"0ae8ca85-0e99-47d2-9693-23a51b043fcb","Type":"ContainerStarted","Data":"7a00a859787293e453face06732acacf18a5008f2631eb120d5ba75bdd065061"} Jul 06 19:11:46.795715 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:46.795627 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" event={"ID":"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd","Type":"ContainerStarted","Data":"93bfe8f48aaec381ca27b4ba76b42511ebbaa87c112bb5bddd41d3eade6b7d0d"} Jul 06 19:11:46.799091 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:46.798953 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-snzfv" event={"ID":"7cf4aab1-3038-40fd-9edf-450ab09a1e84","Type":"ContainerStarted","Data":"48113c226ccb07b497e7afbec85417f0868b9d489cf4a9902feafc38d615a3ad"} Jul 06 19:11:46.801330 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:46.801236 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2cnx7" event={"ID":"b0dba382-5dc7-4c19-bad4-46d9ad717afc","Type":"ContainerStarted","Data":"601b89cbe54fdec68e830c3c43a5d912f13be6bfbff18da4c7dd5389ac3458b3"} Jul 06 19:11:46.805265 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:46.805201 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-454zs" event={"ID":"5f592f36-c320-4ddb-abca-8bc3f6586ae0","Type":"ContainerStarted","Data":"baead8bb81bd3899be9df5864ab77ac4cb0a313703721f16f39091c2e8baa9ee"} Jul 06 19:11:46.820553 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:46.820523 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zcnp6" event={"ID":"a751eaf8-99c8-4e12-92a7-e135a18167a9","Type":"ContainerStarted","Data":"7f438404aae95d90244c17fa4e8f7750b3957f41e48058f043e0172cd38395aa"} Jul 06 19:11:46.827923 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:46.827896 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zlp9z" event={"ID":"bec8c587-08a7-467a-8fdc-cfdc44d0571c","Type":"ContainerStarted","Data":"9f70de4188f2fa4240e220ff7ecc781f63a700e0b1faa278dfa1e4fcd53ee273"} Jul 06 19:11:46.835100 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:46.835071 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" event={"ID":"e36a6168-595f-4a90-a287-3aa5d4c26d6c","Type":"ContainerStarted","Data":"7a21b668b0acb48f0d73afd6e0b3fc19ac099d369402da7fa6c88917786eb7b5"} Jul 06 19:11:46.848308 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:46.848228 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" event={"ID":"a2d615bf-2066-4a10-b984-d7c2be1c8b6d","Type":"ContainerStarted","Data":"092b0430e6e35f47a803d5d9a43031b04308f7894ca2a4dd10ed342c19abb54a"} Jul 06 19:11:47.147151 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:47.147060 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs\") pod \"network-metrics-daemon-2pdkn\" (UID: \"2db677cb-ce90-46dd-b292-3d8c63da4fbb\") " pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:11:47.147339 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:47.147233 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:47.147339 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:47.147298 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs podName:2db677cb-ce90-46dd-b292-3d8c63da4fbb nodeName:}" failed. No retries permitted until 2026-07-06 19:11:49.147278324 +0000 UTC m=+6.043167842 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs") pod "network-metrics-daemon-2pdkn" (UID: "2db677cb-ce90-46dd-b292-3d8c63da4fbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:47.248253 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:47.248216 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret\") pod \"global-pull-secret-syncer-qrqcb\" (UID: \"4240cab6-4ae1-4d08-ae5d-d3040f4f9d61\") " pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:11:47.248452 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:47.248367 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:47.248452 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:47.248430 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret podName:4240cab6-4ae1-4d08-ae5d-d3040f4f9d61 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:49.248411399 +0000 UTC m=+6.144300931 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret") pod "global-pull-secret-syncer-qrqcb" (UID: "4240cab6-4ae1-4d08-ae5d-d3040f4f9d61") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:47.349238 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:47.348537 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5dzh\" (UniqueName: \"kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh\") pod \"network-check-target-zxj86\" (UID: \"953f279d-10c8-4b4c-97e5-a7bd6ecde126\") " pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:11:47.349238 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:47.348789 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:11:47.349238 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:47.348808 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:11:47.349238 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:47.348822 2566 projected.go:196] Error preparing data for projected volume kube-api-access-c5dzh for pod openshift-network-diagnostics/network-check-target-zxj86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:47.349238 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:47.348880 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh podName:953f279d-10c8-4b4c-97e5-a7bd6ecde126 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:49.348860103 +0000 UTC m=+6.244749622 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-c5dzh" (UniqueName: "kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh") pod "network-check-target-zxj86" (UID: "953f279d-10c8-4b4c-97e5-a7bd6ecde126") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:47.594557 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:47.594466 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:06:46 +0000 UTC" deadline="2028-03-05 21:59:38.093716622 +0000 UTC" Jul 06 19:11:47.594557 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:47.594510 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14594h47m50.499211529s" Jul 06 19:11:47.759900 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:47.759866 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:11:47.760085 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:47.760027 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pdkn" podUID="2db677cb-ce90-46dd-b292-3d8c63da4fbb" Jul 06 19:11:47.760465 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:47.760445 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:11:47.760596 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:47.760563 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrqcb" podUID="4240cab6-4ae1-4d08-ae5d-d3040f4f9d61" Jul 06 19:11:48.756986 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:48.756950 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:11:48.757497 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:48.757089 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxj86" podUID="953f279d-10c8-4b4c-97e5-a7bd6ecde126" Jul 06 19:11:49.166046 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:49.165154 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs\") pod \"network-metrics-daemon-2pdkn\" (UID: \"2db677cb-ce90-46dd-b292-3d8c63da4fbb\") " pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:11:49.166046 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:49.165383 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:49.166046 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:49.165451 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs podName:2db677cb-ce90-46dd-b292-3d8c63da4fbb nodeName:}" failed. No retries permitted until 2026-07-06 19:11:53.165429976 +0000 UTC m=+10.061319511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs") pod "network-metrics-daemon-2pdkn" (UID: "2db677cb-ce90-46dd-b292-3d8c63da4fbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:49.267296 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:49.266608 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret\") pod \"global-pull-secret-syncer-qrqcb\" (UID: \"4240cab6-4ae1-4d08-ae5d-d3040f4f9d61\") " pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:11:49.267296 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:49.266806 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:49.267296 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:49.266876 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret podName:4240cab6-4ae1-4d08-ae5d-d3040f4f9d61 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:53.266856644 +0000 UTC m=+10.162746161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret") pod "global-pull-secret-syncer-qrqcb" (UID: "4240cab6-4ae1-4d08-ae5d-d3040f4f9d61") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:49.367925 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:49.367815 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5dzh\" (UniqueName: \"kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh\") pod \"network-check-target-zxj86\" (UID: \"953f279d-10c8-4b4c-97e5-a7bd6ecde126\") " pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:11:49.368101 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:49.367974 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:11:49.368101 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:49.368000 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:11:49.368101 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:49.368013 2566 projected.go:196] Error preparing data for projected volume kube-api-access-c5dzh for pod openshift-network-diagnostics/network-check-target-zxj86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:49.368101 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:49.368101 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh podName:953f279d-10c8-4b4c-97e5-a7bd6ecde126 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:53.368078834 +0000 UTC m=+10.263968367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-c5dzh" (UniqueName: "kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh") pod "network-check-target-zxj86" (UID: "953f279d-10c8-4b4c-97e5-a7bd6ecde126") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:49.757719 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:49.757682 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:11:49.758238 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:49.757859 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrqcb" podUID="4240cab6-4ae1-4d08-ae5d-d3040f4f9d61" Jul 06 19:11:49.758238 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:49.758018 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:11:49.758238 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:49.758152 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pdkn" podUID="2db677cb-ce90-46dd-b292-3d8c63da4fbb" Jul 06 19:11:50.757573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:50.757530 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:11:50.757764 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:50.757683 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxj86" podUID="953f279d-10c8-4b4c-97e5-a7bd6ecde126" Jul 06 19:11:51.761165 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:51.761131 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:11:51.761639 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:51.761186 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:11:51.761639 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:51.761294 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pdkn" podUID="2db677cb-ce90-46dd-b292-3d8c63da4fbb" Jul 06 19:11:51.761639 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:51.761418 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrqcb" podUID="4240cab6-4ae1-4d08-ae5d-d3040f4f9d61" Jul 06 19:11:52.757454 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:52.757420 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:11:52.757621 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:52.757551 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxj86" podUID="953f279d-10c8-4b4c-97e5-a7bd6ecde126" Jul 06 19:11:53.200079 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:53.199449 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs\") pod \"network-metrics-daemon-2pdkn\" (UID: \"2db677cb-ce90-46dd-b292-3d8c63da4fbb\") " pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:11:53.200079 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:53.199648 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:53.200079 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:53.199716 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs podName:2db677cb-ce90-46dd-b292-3d8c63da4fbb nodeName:}" failed. No retries permitted until 2026-07-06 19:12:01.199695529 +0000 UTC m=+18.095585042 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs") pod "network-metrics-daemon-2pdkn" (UID: "2db677cb-ce90-46dd-b292-3d8c63da4fbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:53.300256 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:53.300208 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret\") pod \"global-pull-secret-syncer-qrqcb\" (UID: \"4240cab6-4ae1-4d08-ae5d-d3040f4f9d61\") " pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:11:53.300441 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:53.300422 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:53.300519 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:53.300493 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret podName:4240cab6-4ae1-4d08-ae5d-d3040f4f9d61 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:01.300474948 +0000 UTC m=+18.196364471 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret") pod "global-pull-secret-syncer-qrqcb" (UID: "4240cab6-4ae1-4d08-ae5d-d3040f4f9d61") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:53.401546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:53.401502 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5dzh\" (UniqueName: \"kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh\") pod \"network-check-target-zxj86\" (UID: \"953f279d-10c8-4b4c-97e5-a7bd6ecde126\") " pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:11:53.401778 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:53.401701 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:11:53.401778 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:53.401757 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:11:53.401778 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:53.401770 2566 projected.go:196] Error preparing data for projected volume kube-api-access-c5dzh for pod openshift-network-diagnostics/network-check-target-zxj86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:53.401999 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:53.401837 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh podName:953f279d-10c8-4b4c-97e5-a7bd6ecde126 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:01.401818206 +0000 UTC m=+18.297707720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-c5dzh" (UniqueName: "kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh") pod "network-check-target-zxj86" (UID: "953f279d-10c8-4b4c-97e5-a7bd6ecde126") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:53.757810 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:53.757774 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:11:53.758783 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:53.758739 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrqcb" podUID="4240cab6-4ae1-4d08-ae5d-d3040f4f9d61" Jul 06 19:11:53.758783 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:53.758773 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:11:53.758971 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:53.758865 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pdkn" podUID="2db677cb-ce90-46dd-b292-3d8c63da4fbb" Jul 06 19:11:54.757243 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:54.757205 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:11:54.757713 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:54.757342 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxj86" podUID="953f279d-10c8-4b4c-97e5-a7bd6ecde126" Jul 06 19:11:55.758101 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:55.757989 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:11:55.758567 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:55.758140 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pdkn" podUID="2db677cb-ce90-46dd-b292-3d8c63da4fbb" Jul 06 19:11:55.758567 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:55.757989 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:11:55.758567 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:55.758337 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrqcb" podUID="4240cab6-4ae1-4d08-ae5d-d3040f4f9d61" Jul 06 19:11:56.757947 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:56.757908 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:11:56.758180 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:56.758041 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxj86" podUID="953f279d-10c8-4b4c-97e5-a7bd6ecde126" Jul 06 19:11:57.757416 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:57.757202 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:11:57.757568 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:57.757221 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:11:57.757568 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:57.757480 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pdkn" podUID="2db677cb-ce90-46dd-b292-3d8c63da4fbb" Jul 06 19:11:57.757686 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:57.757561 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrqcb" podUID="4240cab6-4ae1-4d08-ae5d-d3040f4f9d61" Jul 06 19:11:58.757663 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:58.757634 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:11:58.758120 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:58.757783 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxj86" podUID="953f279d-10c8-4b4c-97e5-a7bd6ecde126" Jul 06 19:11:59.757879 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:59.757844 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:11:59.757879 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:11:59.757864 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:11:59.758355 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:59.757984 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pdkn" podUID="2db677cb-ce90-46dd-b292-3d8c63da4fbb" Jul 06 19:11:59.758355 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:11:59.758071 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrqcb" podUID="4240cab6-4ae1-4d08-ae5d-d3040f4f9d61" Jul 06 19:12:00.757803 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:00.757768 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:12:00.758027 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:00.757899 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxj86" podUID="953f279d-10c8-4b4c-97e5-a7bd6ecde126" Jul 06 19:12:01.266473 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:01.266437 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs\") pod \"network-metrics-daemon-2pdkn\" (UID: \"2db677cb-ce90-46dd-b292-3d8c63da4fbb\") " pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:12:01.266666 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:01.266611 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:12:01.266748 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:01.266692 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs podName:2db677cb-ce90-46dd-b292-3d8c63da4fbb nodeName:}" failed. No retries permitted until 2026-07-06 19:12:17.266669312 +0000 UTC m=+34.162558827 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs") pod "network-metrics-daemon-2pdkn" (UID: "2db677cb-ce90-46dd-b292-3d8c63da4fbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:12:01.367793 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:01.367741 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret\") pod \"global-pull-secret-syncer-qrqcb\" (UID: \"4240cab6-4ae1-4d08-ae5d-d3040f4f9d61\") " pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:12:01.367977 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:01.367917 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:12:01.368031 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:01.367994 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret podName:4240cab6-4ae1-4d08-ae5d-d3040f4f9d61 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:17.367977899 +0000 UTC m=+34.263867413 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret") pod "global-pull-secret-syncer-qrqcb" (UID: "4240cab6-4ae1-4d08-ae5d-d3040f4f9d61") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:12:01.468676 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:01.468640 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5dzh\" (UniqueName: \"kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh\") pod \"network-check-target-zxj86\" (UID: \"953f279d-10c8-4b4c-97e5-a7bd6ecde126\") " pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:12:01.468854 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:01.468833 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:12:01.468897 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:01.468858 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:12:01.468897 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:01.468868 2566 projected.go:196] Error preparing data for projected volume kube-api-access-c5dzh for pod openshift-network-diagnostics/network-check-target-zxj86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:12:01.468994 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:01.468922 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh podName:953f279d-10c8-4b4c-97e5-a7bd6ecde126 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:17.468904729 +0000 UTC m=+34.364794263 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-c5dzh" (UniqueName: "kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh") pod "network-check-target-zxj86" (UID: "953f279d-10c8-4b4c-97e5-a7bd6ecde126") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:12:01.760361 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:01.760294 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:12:01.760709 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:01.760394 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pdkn" podUID="2db677cb-ce90-46dd-b292-3d8c63da4fbb" Jul 06 19:12:01.760709 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:01.760419 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:12:01.760709 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:01.760463 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrqcb" podUID="4240cab6-4ae1-4d08-ae5d-d3040f4f9d61" Jul 06 19:12:02.757164 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:02.757125 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:12:02.757366 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:02.757247 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxj86" podUID="953f279d-10c8-4b4c-97e5-a7bd6ecde126" Jul 06 19:12:03.758540 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:03.758500 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:12:03.759045 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:03.758635 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pdkn" podUID="2db677cb-ce90-46dd-b292-3d8c63da4fbb" Jul 06 19:12:03.759045 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:03.758695 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:12:03.759045 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:03.758816 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrqcb" podUID="4240cab6-4ae1-4d08-ae5d-d3040f4f9d61" Jul 06 19:12:04.756989 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:04.756954 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:12:04.757146 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:04.757063 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxj86" podUID="953f279d-10c8-4b4c-97e5-a7bd6ecde126" Jul 06 19:12:05.757812 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:05.757783 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:12:05.758239 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:05.757916 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pdkn" podUID="2db677cb-ce90-46dd-b292-3d8c63da4fbb" Jul 06 19:12:05.758239 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:05.757940 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:12:05.758239 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:05.758040 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrqcb" podUID="4240cab6-4ae1-4d08-ae5d-d3040f4f9d61" Jul 06 19:12:06.757596 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:06.757439 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:12:06.757703 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:06.757684 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxj86" podUID="953f279d-10c8-4b4c-97e5-a7bd6ecde126" Jul 06 19:12:06.895707 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:06.895447 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-201.ec2.internal" event={"ID":"a35ae4d1cb32eff9744780fd38da560f","Type":"ContainerStarted","Data":"447ecab94181ccd0a5b871af7d311ed3fb619286e02ed70a0f0a7e4a9b469d25"} Jul 06 19:12:06.897473 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:06.897430 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zlp9z" event={"ID":"bec8c587-08a7-467a-8fdc-cfdc44d0571c","Type":"ContainerStarted","Data":"b330aba58ee600f53569351e004dcb89a197551d5db3988f4e7246ba4ab63656"} Jul 06 19:12:06.899107 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:06.898982 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" event={"ID":"e36a6168-595f-4a90-a287-3aa5d4c26d6c","Type":"ContainerStarted","Data":"80b7668c6443ee4100876adaa24c86eeaa404c15339e51d3d951ad3f2c1b1fdb"} Jul 06 19:12:06.901163 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:06.901147 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:12:06.901485 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:06.901467 2566 generic.go:356] "Generic (PLEG): container finished" podID="a2d615bf-2066-4a10-b984-d7c2be1c8b6d" containerID="39b3dc7538d2f952b07dfb1d8298167ff3bbe032401595762a67e3786b8f54a9" exitCode=1 Jul 06 19:12:06.901546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:06.901530 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" event={"ID":"a2d615bf-2066-4a10-b984-d7c2be1c8b6d","Type":"ContainerStarted","Data":"d67eba940e06b87ebda3068eaec7623b1ec3e71dc8cf99b36dd60629724e44c8"} Jul 06 19:12:06.901595 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:06.901585 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" event={"ID":"a2d615bf-2066-4a10-b984-d7c2be1c8b6d","Type":"ContainerStarted","Data":"c3eefbd4db94fe5f379096cba492f6869540acf8e75bfb997c74fefdf839ddb1"} Jul 06 19:12:06.901640 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:06.901616 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" event={"ID":"a2d615bf-2066-4a10-b984-d7c2be1c8b6d","Type":"ContainerDied","Data":"39b3dc7538d2f952b07dfb1d8298167ff3bbe032401595762a67e3786b8f54a9"} Jul 06 19:12:06.901680 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:06.901652 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" event={"ID":"a2d615bf-2066-4a10-b984-d7c2be1c8b6d","Type":"ContainerStarted","Data":"1942c8d6c402e55ccc9d4940ff57286ee9112c2cb2320045f5fd4548e25f9320"} Jul 06 19:12:06.902795 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:06.902772 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bfxrw" event={"ID":"0ae8ca85-0e99-47d2-9693-23a51b043fcb","Type":"ContainerStarted","Data":"432bf20766d297b086c72b7b5cfd95675a0a9011f9334a5e987680dc4639cb43"} Jul 06 19:12:06.904001 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:06.903984 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" event={"ID":"02270f78-7b3e-4846-9ee3-9ee6d4cc26fd","Type":"ContainerStarted","Data":"ea1536c8deffed7420e1a573326f26f8752bb5d871f6155bbf7b759d952394e5"} Jul 06 19:12:06.905460 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:06.905442 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal" event={"ID":"c744258162850b158aecb328909531ba","Type":"ContainerStarted","Data":"9859182fee55a4297ba981c731ba680260e4df3216e5b0dec94a680fd7dafde9"} Jul 06 19:12:06.908227 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:06.908200 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2cnx7" event={"ID":"b0dba382-5dc7-4c19-bad4-46d9ad717afc","Type":"ContainerStarted","Data":"bf99483cfed12a90f5c0f6babcdb493203b7f565eb438f22e26fcc7233cd4a68"} Jul 06 19:12:06.910176 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:06.910157 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-454zs" event={"ID":"5f592f36-c320-4ddb-abca-8bc3f6586ae0","Type":"ContainerStarted","Data":"7cf482b0ce6fe76168a72dbf5fdd31ae9f15b2475874a986215e50599eed9f1c"} Jul 06 19:12:06.911375 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:06.911340 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zcnp6" event={"ID":"a751eaf8-99c8-4e12-92a7-e135a18167a9","Type":"ContainerStarted","Data":"44003f37143dfb7c4ba35c62521eb94a3ee9819a7a94253741047d112384f783"} Jul 06 19:12:06.912788 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:06.912754 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-201.ec2.internal" podStartSLOduration=21.912743209 podStartE2EDuration="21.912743209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:06.912507974 +0000 UTC m=+23.808397508" watchObservedRunningTime="2026-07-06 19:12:06.912743209 +0000 UTC m=+23.808632738" Jul 06 19:12:06.963671 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:06.963624 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fkb5l" podStartSLOduration=2.9149478220000002 podStartE2EDuration="22.963606173s" totalImagesPullingTime="20.048658351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:06.963169396 +0000 UTC m=+23.859058930" watchObservedRunningTime="2026-07-06 19:12:06.963606173 +0000 UTC m=+23.859495704" Jul 06 19:12:06.983011 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:06.982967 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zcnp6" podStartSLOduration=2.942790536 podStartE2EDuration="22.982950385s" totalImagesPullingTime="20.040159849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:06.98260954 +0000 UTC m=+23.878499072" watchObservedRunningTime="2026-07-06 19:12:06.982950385 +0000 UTC m=+23.878839917" Jul 06 19:12:07.000633 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.000592 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-454zs" podStartSLOduration=2.965697753 podStartE2EDuration="23.000573195s" totalImagesPullingTime="20.034875442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:07.000533114 +0000 UTC m=+23.896422686" watchObservedRunningTime="2026-07-06 19:12:07.000573195 +0000 UTC m=+23.896462745" Jul 06 19:12:07.049491 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.049291 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2cnx7" podStartSLOduration=3.036035362 podStartE2EDuration="23.049268529s" totalImagesPullingTime="20.013233167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:07.049211786 +0000 UTC m=+23.945101334" watchObservedRunningTime="2026-07-06 19:12:07.049268529 +0000 UTC m=+23.945158064" Jul 06 19:12:07.072615 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.072567 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zlp9z" podStartSLOduration=2.812101363 podStartE2EDuration="23.072549373s" totalImagesPullingTime="20.26044801s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:07.07203451 +0000 UTC m=+23.967924043" watchObservedRunningTime="2026-07-06 19:12:07.072549373 +0000 UTC m=+23.968438926" Jul 06 19:12:07.107597 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.107562 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:07.199662 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.199585 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:07.315367 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.315331 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:07.426060 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.426033 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:07.511138 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.511065 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:07.757292 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.757250 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:12:07.757482 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:07.757381 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrqcb" podUID="4240cab6-4ae1-4d08-ae5d-d3040f4f9d61" Jul 06 19:12:07.757482 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.757432 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:12:07.757598 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:07.757542 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pdkn" podUID="2db677cb-ce90-46dd-b292-3d8c63da4fbb" Jul 06 19:12:07.915961 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.915884 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:12:07.916400 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.916250 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" event={"ID":"a2d615bf-2066-4a10-b984-d7c2be1c8b6d","Type":"ContainerStarted","Data":"204ed321037a9a06b2ec6cc49fc9be40f716c9db09cd506b3f22d738d9698c77"} Jul 06 19:12:07.916400 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.916315 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" event={"ID":"a2d615bf-2066-4a10-b984-d7c2be1c8b6d","Type":"ContainerStarted","Data":"a00eab643faffa89fb8df94f3c9867915068e47d2465f1baa1f59d97c4f1b229"} Jul 06 19:12:07.917590 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.917568 2566 generic.go:356] "Generic (PLEG): container finished" podID="0ae8ca85-0e99-47d2-9693-23a51b043fcb" containerID="432bf20766d297b086c72b7b5cfd95675a0a9011f9334a5e987680dc4639cb43" exitCode=0 Jul 06 19:12:07.917687 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.917627 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bfxrw" event={"ID":"0ae8ca85-0e99-47d2-9693-23a51b043fcb","Type":"ContainerDied","Data":"432bf20766d297b086c72b7b5cfd95675a0a9011f9334a5e987680dc4639cb43"} Jul 06 19:12:07.919180 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.919149 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-snzfv" event={"ID":"7cf4aab1-3038-40fd-9edf-450ab09a1e84","Type":"ContainerStarted","Data":"6f9d0acf12918ea9bb20cf824d14caa47cc36093c75f09439febedcfd254f887"} Jul 06 19:12:07.920891 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.920867 2566 generic.go:356] "Generic (PLEG): container finished" podID="c744258162850b158aecb328909531ba" containerID="9859182fee55a4297ba981c731ba680260e4df3216e5b0dec94a680fd7dafde9" exitCode=0 Jul 06 19:12:07.920983 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.920896 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal" event={"ID":"c744258162850b158aecb328909531ba","Type":"ContainerDied","Data":"9859182fee55a4297ba981c731ba680260e4df3216e5b0dec94a680fd7dafde9"} Jul 06 19:12:07.920983 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.920935 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal" event={"ID":"c744258162850b158aecb328909531ba","Type":"ContainerStarted","Data":"e76d9cd2b5fce446d8397ef5d7c304e562230390390a2eae2350a34c7796d9ed"} Jul 06 19:12:07.960322 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.960262 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-201.ec2.internal" podStartSLOduration=22.960242215 podStartE2EDuration="22.960242215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:07.960117872 +0000 UTC m=+24.856007406" watchObservedRunningTime="2026-07-06 19:12:07.960242215 +0000 UTC m=+24.856131749" Jul 06 19:12:07.981896 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:07.981852 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-snzfv" podStartSLOduration=7.818016926 podStartE2EDuration="23.981836237s" totalImagesPullingTime="16.163819311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:07.981491777 +0000 UTC m=+24.877381309" watchObservedRunningTime="2026-07-06 19:12:07.981836237 +0000 UTC m=+24.877725769" Jul 06 19:12:08.106503 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:08.106466 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2cnx7" Jul 06 19:12:08.107215 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:08.107193 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2cnx7" Jul 06 19:12:08.115185 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:08.115151 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:08.226581 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:08.226513 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:08.334122 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:08.334086 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:08.530591 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:08.530566 2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 19:12:08.621121 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:08.621033 2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T19:12:08.530585063Z","UUID":"4186e04b-b5d9-40d9-ae35-645e264e46af","Handler":null,"Name":"","Endpoint":""} Jul 06 19:12:08.623461 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:08.623441 2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 19:12:08.623581 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:08.623475 2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 19:12:08.757353 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:08.757272 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:12:08.757568 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:08.757379 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxj86" podUID="953f279d-10c8-4b4c-97e5-a7bd6ecde126" Jul 06 19:12:08.925824 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:08.925762 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" event={"ID":"e36a6168-595f-4a90-a287-3aa5d4c26d6c","Type":"ContainerStarted","Data":"d4711a7a916e0c20986178c795d66309865cb9d7038389d8391042f21bcd9175"} Jul 06 19:12:09.388779 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:09.388685 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:09.470731 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:09.470680 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:09.757195 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:09.757115 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:12:09.757195 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:09.757137 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:12:09.757377 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:09.757259 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrqcb" podUID="4240cab6-4ae1-4d08-ae5d-d3040f4f9d61" Jul 06 19:12:09.757377 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:09.757357 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pdkn" podUID="2db677cb-ce90-46dd-b292-3d8c63da4fbb" Jul 06 19:12:09.931079 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:09.930936 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:12:09.931463 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:09.931451 2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 06 19:12:09.931514 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:09.931475 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" event={"ID":"a2d615bf-2066-4a10-b984-d7c2be1c8b6d","Type":"ContainerStarted","Data":"c99dd09ab208f330518ba666ab5c180a4cf8899934d5d99aad5e81ea780a3604"} Jul 06 19:12:10.757431 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:10.757397 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:12:10.757606 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:10.757506 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxj86" podUID="953f279d-10c8-4b4c-97e5-a7bd6ecde126" Jul 06 19:12:10.935521 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:10.935485 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" event={"ID":"e36a6168-595f-4a90-a287-3aa5d4c26d6c","Type":"ContainerStarted","Data":"f78273c7a7426bf8fc109b5e3624cb75bfbe86b21b600e7b463e5faf6f86d824"} Jul 06 19:12:10.955884 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:10.955830 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l4v" podStartSLOduration=4.112837265 podStartE2EDuration="26.955814855s" totalImagesPullingTime="22.84297759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:10.955358571 +0000 UTC m=+27.851248100" watchObservedRunningTime="2026-07-06 19:12:10.955814855 +0000 UTC m=+27.851704388" Jul 06 19:12:11.757349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:11.757259 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:12:11.757499 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:11.757266 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:12:11.757499 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:11.757384 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrqcb" podUID="4240cab6-4ae1-4d08-ae5d-d3040f4f9d61" Jul 06 19:12:11.757499 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:11.757438 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pdkn" podUID="2db677cb-ce90-46dd-b292-3d8c63da4fbb" Jul 06 19:12:11.940520 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:11.940490 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:12:11.941328 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:11.940831 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" event={"ID":"a2d615bf-2066-4a10-b984-d7c2be1c8b6d","Type":"ContainerStarted","Data":"b943594a7bcd2a770f9e6ab7cae2106b7208bea0a994058ced28dc4494eac858"} Jul 06 19:12:11.941404 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:11.941388 2566 scope.go:117] "RemoveContainer" containerID="39b3dc7538d2f952b07dfb1d8298167ff3bbe032401595762a67e3786b8f54a9" Jul 06 19:12:12.757259 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:12.757225 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:12:12.757489 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:12.757352 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxj86" podUID="953f279d-10c8-4b4c-97e5-a7bd6ecde126" Jul 06 19:12:12.947554 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:12.947522 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:12:12.947981 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:12.947957 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" event={"ID":"a2d615bf-2066-4a10-b984-d7c2be1c8b6d","Type":"ContainerStarted","Data":"b1d63a714268a963e9af7b63475141aa89d047cabda5b21c8c1f7f952a87104e"} Jul 06 19:12:12.948241 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:12.948219 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:12:12.948470 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:12.948429 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:12:12.948668 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:12.948639 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:12:12.966031 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:12.965999 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:12:12.966405 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:12.966387 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:12:12.980442 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:12.980380 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" podStartSLOduration=8.619607098 podStartE2EDuration="28.98036307s" totalImagesPullingTime="20.360755972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:12.979274414 +0000 UTC m=+29.875163985" watchObservedRunningTime="2026-07-06 19:12:12.98036307 +0000 UTC m=+29.876252602" Jul 06 19:12:13.382811 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:13.382624 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zxj86"] Jul 06 19:12:13.382811 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:13.382761 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:12:13.383066 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:13.382861 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxj86" podUID="953f279d-10c8-4b4c-97e5-a7bd6ecde126" Jul 06 19:12:13.386023 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:13.385994 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qrqcb"] Jul 06 19:12:13.386167 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:13.386096 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:12:13.386233 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:13.386190 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrqcb" podUID="4240cab6-4ae1-4d08-ae5d-d3040f4f9d61" Jul 06 19:12:13.386711 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:13.386667 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2pdkn"] Jul 06 19:12:13.386872 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:13.386789 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:12:13.386941 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:13.386878 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pdkn" podUID="2db677cb-ce90-46dd-b292-3d8c63da4fbb" Jul 06 19:12:14.469885 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:14.469854 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:14.574780 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:14.574683 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:14.750730 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:14.750701 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2cnx7" Jul 06 19:12:14.750900 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:14.750872 2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 06 19:12:14.751280 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:14.751257 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2cnx7" Jul 06 19:12:14.757768 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:14.757745 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:12:14.757878 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:14.757834 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxj86" podUID="953f279d-10c8-4b4c-97e5-a7bd6ecde126" Jul 06 19:12:14.842878 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:14.842799 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:14.953528 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:14.953496 2566 generic.go:356] "Generic (PLEG): container finished" podID="0ae8ca85-0e99-47d2-9693-23a51b043fcb" containerID="cd10427c4d0740ea8726d7e51827cb642e3ea7af7f1b7d98abeb93745bc70694" exitCode=0 Jul 06 19:12:14.953693 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:14.953579 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bfxrw" event={"ID":"0ae8ca85-0e99-47d2-9693-23a51b043fcb","Type":"ContainerDied","Data":"cd10427c4d0740ea8726d7e51827cb642e3ea7af7f1b7d98abeb93745bc70694"} Jul 06 19:12:15.758090 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:15.758017 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:12:15.758535 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:15.758153 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pdkn" podUID="2db677cb-ce90-46dd-b292-3d8c63da4fbb" Jul 06 19:12:15.758535 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:15.758017 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:12:15.758535 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:15.758229 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrqcb" podUID="4240cab6-4ae1-4d08-ae5d-d3040f4f9d61" Jul 06 19:12:16.757830 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:16.757647 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:12:16.758019 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:16.757916 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxj86" podUID="953f279d-10c8-4b4c-97e5-a7bd6ecde126" Jul 06 19:12:16.963005 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:16.962971 2566 generic.go:356] "Generic (PLEG): container finished" podID="0ae8ca85-0e99-47d2-9693-23a51b043fcb" containerID="e3bb1371fd933e4f5c27aece62743a5dedd865317352a20e33edc92c71bc6dc4" exitCode=0 Jul 06 19:12:16.963428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:16.963023 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bfxrw" event={"ID":"0ae8ca85-0e99-47d2-9693-23a51b043fcb","Type":"ContainerDied","Data":"e3bb1371fd933e4f5c27aece62743a5dedd865317352a20e33edc92c71bc6dc4"} Jul 06 19:12:17.289969 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:17.289934 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs\") pod \"network-metrics-daemon-2pdkn\" (UID: \"2db677cb-ce90-46dd-b292-3d8c63da4fbb\") " pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:12:17.290126 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:17.290086 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:12:17.290163 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:17.290152 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs podName:2db677cb-ce90-46dd-b292-3d8c63da4fbb nodeName:}" failed. No retries permitted until 2026-07-06 19:12:49.290135779 +0000 UTC m=+66.186025290 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs") pod "network-metrics-daemon-2pdkn" (UID: "2db677cb-ce90-46dd-b292-3d8c63da4fbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:12:17.370408 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:17.370377 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:17.390511 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:17.390480 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret\") pod \"global-pull-secret-syncer-qrqcb\" (UID: \"4240cab6-4ae1-4d08-ae5d-d3040f4f9d61\") " pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:12:17.390607 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:17.390588 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:12:17.390659 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:17.390650 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret podName:4240cab6-4ae1-4d08-ae5d-d3040f4f9d61 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:49.390623587 +0000 UTC m=+66.286513097 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret") pod "global-pull-secret-syncer-qrqcb" (UID: "4240cab6-4ae1-4d08-ae5d-d3040f4f9d61") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:12:17.466059 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:17.466023 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:17.491485 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:17.491455 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5dzh\" (UniqueName: \"kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh\") pod \"network-check-target-zxj86\" (UID: \"953f279d-10c8-4b4c-97e5-a7bd6ecde126\") " pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:12:17.491628 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:17.491574 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:12:17.491628 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:17.491592 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:12:17.491628 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:17.491601 2566 projected.go:196] Error preparing data for projected volume kube-api-access-c5dzh for pod openshift-network-diagnostics/network-check-target-zxj86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:12:17.491743 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:17.491653 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh podName:953f279d-10c8-4b4c-97e5-a7bd6ecde126 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:49.491635008 +0000 UTC m=+66.387524536 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-c5dzh" (UniqueName: "kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh") pod "network-check-target-zxj86" (UID: "953f279d-10c8-4b4c-97e5-a7bd6ecde126") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:12:17.757273 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:17.757191 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:12:17.757420 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:17.757295 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrqcb" podUID="4240cab6-4ae1-4d08-ae5d-d3040f4f9d61" Jul 06 19:12:17.757420 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:17.757354 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:12:17.757487 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:17.757447 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pdkn" podUID="2db677cb-ce90-46dd-b292-3d8c63da4fbb" Jul 06 19:12:18.757622 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:18.757591 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:12:18.758056 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:18.757687 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxj86" podUID="953f279d-10c8-4b4c-97e5-a7bd6ecde126" Jul 06 19:12:18.970349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:18.970318 2566 generic.go:356] "Generic (PLEG): container finished" podID="0ae8ca85-0e99-47d2-9693-23a51b043fcb" containerID="1a3fc24c2a4958ad4781c2e915f505974ad8ef2ea8d647a88170637dd6993eaa" exitCode=0 Jul 06 19:12:18.970533 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:18.970369 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bfxrw" event={"ID":"0ae8ca85-0e99-47d2-9693-23a51b043fcb","Type":"ContainerDied","Data":"1a3fc24c2a4958ad4781c2e915f505974ad8ef2ea8d647a88170637dd6993eaa"} Jul 06 19:12:19.279609 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.279575 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:19.757166 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.757129 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:12:19.757368 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:19.757264 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrqcb" podUID="4240cab6-4ae1-4d08-ae5d-d3040f4f9d61" Jul 06 19:12:19.757368 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.757318 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:12:19.757478 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:19.757418 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pdkn" podUID="2db677cb-ce90-46dd-b292-3d8c63da4fbb" Jul 06 19:12:19.766254 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.766230 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-201.ec2.internal" event="NodeReady" Jul 06 19:12:19.766669 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.766350 2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 19:12:19.821114 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.821074 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-62xmt"] Jul 06 19:12:19.830599 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.830574 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mk4bf"] Jul 06 19:12:19.830716 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.830698 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-62xmt" Jul 06 19:12:19.833412 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.833390 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c8xcq\"" Jul 06 19:12:19.833734 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.833705 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 19:12:19.833837 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.833705 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 19:12:19.842416 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.842394 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mk4bf"] Jul 06 19:12:19.842519 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.842429 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-62xmt"] Jul 06 19:12:19.842579 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.842527 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mk4bf" Jul 06 19:12:19.845309 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.845289 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 19:12:19.845432 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.845333 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zsvz6\"" Jul 06 19:12:19.845432 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.845394 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 19:12:19.845560 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.845452 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 19:12:19.911165 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.911128 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvlsx\" (UniqueName: \"kubernetes.io/projected/bca4b412-168b-48a3-8251-28aea09c3427-kube-api-access-vvlsx\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:12:19.911345 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.911190 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca4b412-168b-48a3-8251-28aea09c3427-tmp-dir\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:12:19.911345 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.911251 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bca4b412-168b-48a3-8251-28aea09c3427-config-volume\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:12:19.911345 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.911331 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:12:19.952183 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:19.952105 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:20.011905 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.011864 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bca4b412-168b-48a3-8251-28aea09c3427-config-volume\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:12:20.012124 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.011926 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:12:20.012124 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.011962 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvlsx\" (UniqueName: \"kubernetes.io/projected/bca4b412-168b-48a3-8251-28aea09c3427-kube-api-access-vvlsx\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:12:20.012124 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.011988 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z78bl\" (UniqueName: \"kubernetes.io/projected/3831b65a-0a72-4c06-b6d4-b3bff61a5a32-kube-api-access-z78bl\") pod \"ingress-canary-mk4bf\" (UID: \"3831b65a-0a72-4c06-b6d4-b3bff61a5a32\") " pod="openshift-ingress-canary/ingress-canary-mk4bf" Jul 06 19:12:20.012124 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.012032 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca4b412-168b-48a3-8251-28aea09c3427-tmp-dir\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:12:20.012124 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.012076 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3831b65a-0a72-4c06-b6d4-b3bff61a5a32-cert\") pod \"ingress-canary-mk4bf\" (UID: \"3831b65a-0a72-4c06-b6d4-b3bff61a5a32\") " pod="openshift-ingress-canary/ingress-canary-mk4bf" Jul 06 19:12:20.012124 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:20.012107 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:20.012449 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:20.012207 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls podName:bca4b412-168b-48a3-8251-28aea09c3427 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:20.512188536 +0000 UTC m=+37.408078046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls") pod "dns-default-62xmt" (UID: "bca4b412-168b-48a3-8251-28aea09c3427") : secret "dns-default-metrics-tls" not found Jul 06 19:12:20.012449 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.012381 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca4b412-168b-48a3-8251-28aea09c3427-tmp-dir\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:12:20.012522 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.012490 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bca4b412-168b-48a3-8251-28aea09c3427-config-volume\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:12:20.021795 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.021767 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:20.025274 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.025249 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvlsx\" (UniqueName: \"kubernetes.io/projected/bca4b412-168b-48a3-8251-28aea09c3427-kube-api-access-vvlsx\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:12:20.112964 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.112923 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3831b65a-0a72-4c06-b6d4-b3bff61a5a32-cert\") pod \"ingress-canary-mk4bf\" (UID: \"3831b65a-0a72-4c06-b6d4-b3bff61a5a32\") " pod="openshift-ingress-canary/ingress-canary-mk4bf" Jul 06 19:12:20.113171 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.113015 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z78bl\" (UniqueName: \"kubernetes.io/projected/3831b65a-0a72-4c06-b6d4-b3bff61a5a32-kube-api-access-z78bl\") pod \"ingress-canary-mk4bf\" (UID: \"3831b65a-0a72-4c06-b6d4-b3bff61a5a32\") " pod="openshift-ingress-canary/ingress-canary-mk4bf" Jul 06 19:12:20.115562 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.115536 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3831b65a-0a72-4c06-b6d4-b3bff61a5a32-cert\") pod \"ingress-canary-mk4bf\" (UID: \"3831b65a-0a72-4c06-b6d4-b3bff61a5a32\") " pod="openshift-ingress-canary/ingress-canary-mk4bf" Jul 06 19:12:20.122133 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.122103 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z78bl\" (UniqueName: \"kubernetes.io/projected/3831b65a-0a72-4c06-b6d4-b3bff61a5a32-kube-api-access-z78bl\") pod \"ingress-canary-mk4bf\" (UID: \"3831b65a-0a72-4c06-b6d4-b3bff61a5a32\") " pod="openshift-ingress-canary/ingress-canary-mk4bf" Jul 06 19:12:20.150198 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.150166 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mk4bf" Jul 06 19:12:20.337593 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.337407 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mk4bf"] Jul 06 19:12:20.341215 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:12:20.341182 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3831b65a_0a72_4c06_b6d4_b3bff61a5a32.slice/crio-343daa1d6f02746e46ce83abb41489eb0258ec6546d91edcca5ef08cd1df2dbf WatchSource:0}: Error finding container 343daa1d6f02746e46ce83abb41489eb0258ec6546d91edcca5ef08cd1df2dbf: Status 404 returned error can't find the container with id 343daa1d6f02746e46ce83abb41489eb0258ec6546d91edcca5ef08cd1df2dbf Jul 06 19:12:20.515582 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.515480 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:12:20.515809 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:20.515631 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:20.515809 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:20.515702 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls podName:bca4b412-168b-48a3-8251-28aea09c3427 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:21.515686896 +0000 UTC m=+38.411576406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls") pod "dns-default-62xmt" (UID: "bca4b412-168b-48a3-8251-28aea09c3427") : secret "dns-default-metrics-tls" not found Jul 06 19:12:20.757236 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.757202 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:12:20.760046 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.760025 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-422bs\"" Jul 06 19:12:20.760164 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.760030 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:12:20.760928 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.760915 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:12:20.976524 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:20.976486 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mk4bf" event={"ID":"3831b65a-0a72-4c06-b6d4-b3bff61a5a32","Type":"ContainerStarted","Data":"343daa1d6f02746e46ce83abb41489eb0258ec6546d91edcca5ef08cd1df2dbf"} Jul 06 19:12:21.522569 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:21.522530 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:12:21.522771 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:21.522649 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:21.522771 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:21.522702 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls podName:bca4b412-168b-48a3-8251-28aea09c3427 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:23.522688298 +0000 UTC m=+40.418577808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls") pod "dns-default-62xmt" (UID: "bca4b412-168b-48a3-8251-28aea09c3427") : secret "dns-default-metrics-tls" not found Jul 06 19:12:21.757892 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:21.757818 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:12:21.758335 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:21.758088 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:12:21.760757 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:21.760692 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:12:21.766615 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:21.762516 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r97mf\"" Jul 06 19:12:21.766615 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:21.762692 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:12:22.463663 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:22.463630 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:22.526269 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:22.526234 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:23.518674 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:23.518635 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:23.538738 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:23.538694 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:12:23.538914 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:23.538881 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:23.538978 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:23.538957 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls podName:bca4b412-168b-48a3-8251-28aea09c3427 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:27.538935141 +0000 UTC m=+44.434824653 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls") pod "dns-default-62xmt" (UID: "bca4b412-168b-48a3-8251-28aea09c3427") : secret "dns-default-metrics-tls" not found Jul 06 19:12:23.917349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:23.917314 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:24.344045 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:24.343962 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:24.401225 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:24.401189 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:25.989489 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:25.989454 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mk4bf" event={"ID":"3831b65a-0a72-4c06-b6d4-b3bff61a5a32","Type":"ContainerStarted","Data":"c833cfa6dfd73fe567dd17d23b57f1da9cc8336cd7741e91ae97fda27e3dee89"} Jul 06 19:12:25.991938 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:25.991917 2566 generic.go:356] "Generic (PLEG): container finished" podID="0ae8ca85-0e99-47d2-9693-23a51b043fcb" containerID="1ed8e66a966a7a7e019a7953757015aaac8858d9076a31c56772edf3a104c3b8" exitCode=0 Jul 06 19:12:25.992056 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:25.991945 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bfxrw" event={"ID":"0ae8ca85-0e99-47d2-9693-23a51b043fcb","Type":"ContainerDied","Data":"1ed8e66a966a7a7e019a7953757015aaac8858d9076a31c56772edf3a104c3b8"} Jul 06 19:12:26.010294 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:26.010242 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mk4bf" podStartSLOduration=1.787051864 podStartE2EDuration="7.010227409s" totalImagesPullingTime="5.223175545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:26.009175073 +0000 UTC m=+42.905064605" watchObservedRunningTime="2026-07-06 19:12:26.010227409 +0000 UTC m=+42.906116931" Jul 06 19:12:26.997507 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:26.997473 2566 generic.go:356] "Generic (PLEG): container finished" podID="0ae8ca85-0e99-47d2-9693-23a51b043fcb" containerID="b9a235be893ffa42bc4a258ce7425840346f11ed7a753a0cdbb08649f950271b" exitCode=0 Jul 06 19:12:26.997915 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:26.997564 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bfxrw" event={"ID":"0ae8ca85-0e99-47d2-9693-23a51b043fcb","Type":"ContainerDied","Data":"b9a235be893ffa42bc4a258ce7425840346f11ed7a753a0cdbb08649f950271b"} Jul 06 19:12:27.565616 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:27.565585 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:12:27.565810 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:27.565704 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:27.565810 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:27.565791 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls podName:bca4b412-168b-48a3-8251-28aea09c3427 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:35.565775747 +0000 UTC m=+52.461665257 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls") pod "dns-default-62xmt" (UID: "bca4b412-168b-48a3-8251-28aea09c3427") : secret "dns-default-metrics-tls" not found Jul 06 19:12:28.003071 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:28.003038 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bfxrw" event={"ID":"0ae8ca85-0e99-47d2-9693-23a51b043fcb","Type":"ContainerStarted","Data":"b4e4cfb84c2bfa28ce563db9fad0f39ac3f731b57a2a342b6ad505da7d109b9d"} Jul 06 19:12:28.032734 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:28.032671 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bfxrw" podStartSLOduration=8.907577884 podStartE2EDuration="44.032653962s" totalImagesPullingTime="35.125076078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:28.032359781 +0000 UTC m=+44.928249314" watchObservedRunningTime="2026-07-06 19:12:28.032653962 +0000 UTC m=+44.928543494" Jul 06 19:12:28.159603 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:28.159553 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:28.233870 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:28.233837 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:31.398302 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:31.398271 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:31.540141 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:31.540106 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:31.616976 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:31.616947 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:35.622664 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:35.622630 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:12:35.623047 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:35.622788 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:35.623047 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:35.622842 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls podName:bca4b412-168b-48a3-8251-28aea09c3427 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:51.622826133 +0000 UTC m=+68.518715647 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls") pod "dns-default-62xmt" (UID: "bca4b412-168b-48a3-8251-28aea09c3427") : secret "dns-default-metrics-tls" not found Jul 06 19:12:38.075806 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:38.075769 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:40.531268 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:40.531231 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:40.635397 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:40.635363 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:44.964155 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:44.964128 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2mdkh" Jul 06 19:12:45.237565 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:45.237461 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:45.317960 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:45.317922 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:49.324406 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:49.324355 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs\") pod \"network-metrics-daemon-2pdkn\" (UID: \"2db677cb-ce90-46dd-b292-3d8c63da4fbb\") " pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:12:49.327336 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:49.327320 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:12:49.335445 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:49.335429 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 19:12:49.335528 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:49.335482 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs podName:2db677cb-ce90-46dd-b292-3d8c63da4fbb nodeName:}" failed. No retries permitted until 2026-07-06 19:13:53.335466708 +0000 UTC m=+130.231356218 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs") pod "network-metrics-daemon-2pdkn" (UID: "2db677cb-ce90-46dd-b292-3d8c63da4fbb") : secret "metrics-daemon-secret" not found Jul 06 19:12:49.424937 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:49.424894 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret\") pod \"global-pull-secret-syncer-qrqcb\" (UID: \"4240cab6-4ae1-4d08-ae5d-d3040f4f9d61\") " pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:12:49.427995 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:49.427979 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:12:49.438331 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:49.438310 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4240cab6-4ae1-4d08-ae5d-d3040f4f9d61-original-pull-secret\") pod \"global-pull-secret-syncer-qrqcb\" (UID: \"4240cab6-4ae1-4d08-ae5d-d3040f4f9d61\") " pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:12:49.525438 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:49.525405 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5dzh\" (UniqueName: \"kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh\") pod \"network-check-target-zxj86\" (UID: \"953f279d-10c8-4b4c-97e5-a7bd6ecde126\") " pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:12:49.528281 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:49.528262 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:12:49.538969 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:49.538952 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:12:49.549126 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:49.549103 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5dzh\" (UniqueName: \"kubernetes.io/projected/953f279d-10c8-4b4c-97e5-a7bd6ecde126-kube-api-access-c5dzh\") pod \"network-check-target-zxj86\" (UID: \"953f279d-10c8-4b4c-97e5-a7bd6ecde126\") " pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:12:49.569314 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:49.569291 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-422bs\"" Jul 06 19:12:49.577410 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:49.577358 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:12:49.685141 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:49.685113 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrqcb" Jul 06 19:12:49.694120 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:49.694097 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zxj86"] Jul 06 19:12:49.698362 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:12:49.698337 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953f279d_10c8_4b4c_97e5_a7bd6ecde126.slice/crio-d5f520c835cc2cb4f2b9a4be525c415ae36cf95d9dab8a2249fc292dcc713a8f WatchSource:0}: Error finding container d5f520c835cc2cb4f2b9a4be525c415ae36cf95d9dab8a2249fc292dcc713a8f: Status 404 returned error can't find the container with id d5f520c835cc2cb4f2b9a4be525c415ae36cf95d9dab8a2249fc292dcc713a8f Jul 06 19:12:49.807393 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:49.807364 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qrqcb"] Jul 06 19:12:49.810503 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:12:49.810478 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4240cab6_4ae1_4d08_ae5d_d3040f4f9d61.slice/crio-754b23b4a31ab4cd795a026526efd605438bc8cf0af85f83bb1ded59c6ac3187 WatchSource:0}: Error finding container 754b23b4a31ab4cd795a026526efd605438bc8cf0af85f83bb1ded59c6ac3187: Status 404 returned error can't find the container with id 754b23b4a31ab4cd795a026526efd605438bc8cf0af85f83bb1ded59c6ac3187 Jul 06 19:12:50.053172 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:50.053135 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qrqcb" event={"ID":"4240cab6-4ae1-4d08-ae5d-d3040f4f9d61","Type":"ContainerStarted","Data":"754b23b4a31ab4cd795a026526efd605438bc8cf0af85f83bb1ded59c6ac3187"} Jul 06 19:12:50.054117 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:50.054094 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zxj86" event={"ID":"953f279d-10c8-4b4c-97e5-a7bd6ecde126","Type":"ContainerStarted","Data":"d5f520c835cc2cb4f2b9a4be525c415ae36cf95d9dab8a2249fc292dcc713a8f"} Jul 06 19:12:51.640191 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:51.640156 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:12:51.640571 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:51.640321 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:51.640571 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:12:51.640393 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls podName:bca4b412-168b-48a3-8251-28aea09c3427 nodeName:}" failed. No retries permitted until 2026-07-06 19:13:23.64037255 +0000 UTC m=+100.536262063 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls") pod "dns-default-62xmt" (UID: "bca4b412-168b-48a3-8251-28aea09c3427") : secret "dns-default-metrics-tls" not found Jul 06 19:12:53.063977 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:53.063939 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zxj86" event={"ID":"953f279d-10c8-4b4c-97e5-a7bd6ecde126","Type":"ContainerStarted","Data":"0935dbd7b63e6f46294d9e619a392166f8668da804f8e3c301166e2b4600ba6b"} Jul 06 19:12:53.064388 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:53.064028 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:12:53.081409 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:53.081364 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zxj86" podStartSLOduration=65.801561879 podStartE2EDuration="1m9.081350516s" totalImagesPullingTime="3.279788637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:53.080892646 +0000 UTC m=+69.976782180" watchObservedRunningTime="2026-07-06 19:12:53.081350516 +0000 UTC m=+69.977240048" Jul 06 19:12:55.070600 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:55.070516 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qrqcb" event={"ID":"4240cab6-4ae1-4d08-ae5d-d3040f4f9d61","Type":"ContainerStarted","Data":"8ede8e236b19b750ad946022f80bd8edcc1486abb0516c8819b91a7d62866835"} Jul 06 19:12:55.088649 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:12:55.088578 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qrqcb" podStartSLOduration=66.089175347 podStartE2EDuration="1m11.088558697s" totalImagesPullingTime="4.99938335s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:55.088280783 +0000 UTC m=+71.984170317" watchObservedRunningTime="2026-07-06 19:12:55.088558697 +0000 UTC m=+71.984448229" Jul 06 19:13:15.951934 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:15.951825 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:13:16.051848 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:16.051818 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:13:18.024070 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.024033 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7"] Jul 06 19:13:18.026201 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.026175 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7" Jul 06 19:13:18.031713 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.031686 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-2tzs2\"" Jul 06 19:13:18.031850 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.031687 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 19:13:18.031850 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.031690 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 19:13:18.031850 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.031696 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 06 19:13:18.032817 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.032793 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 19:13:18.042310 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.042289 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7"] Jul 06 19:13:18.080193 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.080164 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq"] Jul 06 19:13:18.082097 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.082083 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:13:18.084636 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.084613 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 06 19:13:18.084863 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.084840 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 06 19:13:18.084980 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.084915 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 06 19:13:18.085034 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.084998 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 06 19:13:18.104393 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.104371 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq"] Jul 06 19:13:18.126488 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.126464 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8b1f7b1f-070b-4c96-928e-ec47bf2dc8de-ca\") pod \"cluster-proxy-proxy-agent-864494d4c7-t4mfq\" (UID: \"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:13:18.126649 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.126509 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8b1f7b1f-070b-4c96-928e-ec47bf2dc8de-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-864494d4c7-t4mfq\" (UID: \"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:13:18.126649 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.126535 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8b1f7b1f-070b-4c96-928e-ec47bf2dc8de-hub\") pod \"cluster-proxy-proxy-agent-864494d4c7-t4mfq\" (UID: \"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:13:18.126649 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.126589 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/29fdffb8-755c-420f-91cb-eee58f087733-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7\" (UID: \"29fdffb8-755c-420f-91cb-eee58f087733\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7" Jul 06 19:13:18.126649 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.126628 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b1f7b1f-070b-4c96-928e-ec47bf2dc8de-additional-service-ca\") pod \"cluster-proxy-proxy-agent-864494d4c7-t4mfq\" (UID: \"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:13:18.126894 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.126660 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnbfz\" (UniqueName: \"kubernetes.io/projected/29fdffb8-755c-420f-91cb-eee58f087733-kube-api-access-xnbfz\") pod \"managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7\" (UID: \"29fdffb8-755c-420f-91cb-eee58f087733\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7" Jul 06 19:13:18.126894 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.126691 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8b1f7b1f-070b-4c96-928e-ec47bf2dc8de-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-864494d4c7-t4mfq\" (UID: \"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:13:18.126894 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.126715 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5md2t\" (UniqueName: \"kubernetes.io/projected/8b1f7b1f-070b-4c96-928e-ec47bf2dc8de-kube-api-access-5md2t\") pod \"cluster-proxy-proxy-agent-864494d4c7-t4mfq\" (UID: \"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:13:18.227356 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.227316 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8b1f7b1f-070b-4c96-928e-ec47bf2dc8de-hub\") pod \"cluster-proxy-proxy-agent-864494d4c7-t4mfq\" (UID: \"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:13:18.227356 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.227354 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/29fdffb8-755c-420f-91cb-eee58f087733-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7\" (UID: \"29fdffb8-755c-420f-91cb-eee58f087733\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7" Jul 06 19:13:18.227590 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.227373 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b1f7b1f-070b-4c96-928e-ec47bf2dc8de-additional-service-ca\") pod \"cluster-proxy-proxy-agent-864494d4c7-t4mfq\" (UID: \"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:13:18.227590 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.227404 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnbfz\" (UniqueName: \"kubernetes.io/projected/29fdffb8-755c-420f-91cb-eee58f087733-kube-api-access-xnbfz\") pod \"managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7\" (UID: \"29fdffb8-755c-420f-91cb-eee58f087733\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7" Jul 06 19:13:18.227590 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.227433 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8b1f7b1f-070b-4c96-928e-ec47bf2dc8de-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-864494d4c7-t4mfq\" (UID: \"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:13:18.227590 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.227458 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5md2t\" (UniqueName: \"kubernetes.io/projected/8b1f7b1f-070b-4c96-928e-ec47bf2dc8de-kube-api-access-5md2t\") pod \"cluster-proxy-proxy-agent-864494d4c7-t4mfq\" (UID: \"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:13:18.227590 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.227494 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8b1f7b1f-070b-4c96-928e-ec47bf2dc8de-ca\") pod \"cluster-proxy-proxy-agent-864494d4c7-t4mfq\" (UID: \"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:13:18.227590 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.227545 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8b1f7b1f-070b-4c96-928e-ec47bf2dc8de-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-864494d4c7-t4mfq\" (UID: \"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:13:18.228224 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.228193 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b1f7b1f-070b-4c96-928e-ec47bf2dc8de-additional-service-ca\") pod \"cluster-proxy-proxy-agent-864494d4c7-t4mfq\" (UID: \"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:13:18.230084 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.230062 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8b1f7b1f-070b-4c96-928e-ec47bf2dc8de-ca\") pod \"cluster-proxy-proxy-agent-864494d4c7-t4mfq\" (UID: \"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:13:18.230190 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.230082 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8b1f7b1f-070b-4c96-928e-ec47bf2dc8de-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-864494d4c7-t4mfq\" (UID: \"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:13:18.230190 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.230092 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8b1f7b1f-070b-4c96-928e-ec47bf2dc8de-hub\") pod \"cluster-proxy-proxy-agent-864494d4c7-t4mfq\" (UID: \"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:13:18.230505 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.230481 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/29fdffb8-755c-420f-91cb-eee58f087733-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7\" (UID: \"29fdffb8-755c-420f-91cb-eee58f087733\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7" Jul 06 19:13:18.230556 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.230541 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8b1f7b1f-070b-4c96-928e-ec47bf2dc8de-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-864494d4c7-t4mfq\" (UID: \"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:13:18.244987 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.244964 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnbfz\" (UniqueName: \"kubernetes.io/projected/29fdffb8-755c-420f-91cb-eee58f087733-kube-api-access-xnbfz\") pod \"managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7\" (UID: \"29fdffb8-755c-420f-91cb-eee58f087733\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7" Jul 06 19:13:18.245195 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.245174 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5md2t\" (UniqueName: \"kubernetes.io/projected/8b1f7b1f-070b-4c96-928e-ec47bf2dc8de-kube-api-access-5md2t\") pod \"cluster-proxy-proxy-agent-864494d4c7-t4mfq\" (UID: \"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:13:18.347373 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.347272 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7" Jul 06 19:13:18.390908 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.390850 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:13:18.488329 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.488300 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7"] Jul 06 19:13:18.491654 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:13:18.491625 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29fdffb8_755c_420f_91cb_eee58f087733.slice/crio-15e895cbbc922873531edb4e79a6c797e48577b34c9adbc5ec4cce005ee29de7 WatchSource:0}: Error finding container 15e895cbbc922873531edb4e79a6c797e48577b34c9adbc5ec4cce005ee29de7: Status 404 returned error can't find the container with id 15e895cbbc922873531edb4e79a6c797e48577b34c9adbc5ec4cce005ee29de7 Jul 06 19:13:18.528836 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:18.528805 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq"] Jul 06 19:13:18.532449 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:13:18.532427 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b1f7b1f_070b_4c96_928e_ec47bf2dc8de.slice/crio-aad218fb98513075ea1a5e03ba5351b88cb1eeb01a8d1c489e238682ef2aed93 WatchSource:0}: Error finding container aad218fb98513075ea1a5e03ba5351b88cb1eeb01a8d1c489e238682ef2aed93: Status 404 returned error can't find the container with id aad218fb98513075ea1a5e03ba5351b88cb1eeb01a8d1c489e238682ef2aed93 Jul 06 19:13:19.131989 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:19.131952 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" event={"ID":"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de","Type":"ContainerStarted","Data":"aad218fb98513075ea1a5e03ba5351b88cb1eeb01a8d1c489e238682ef2aed93"} Jul 06 19:13:19.133045 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:19.133022 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7" event={"ID":"29fdffb8-755c-420f-91cb-eee58f087733","Type":"ContainerStarted","Data":"15e895cbbc922873531edb4e79a6c797e48577b34c9adbc5ec4cce005ee29de7"} Jul 06 19:13:23.144757 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:23.144711 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7" event={"ID":"29fdffb8-755c-420f-91cb-eee58f087733","Type":"ContainerStarted","Data":"04d7a85c395c802cccc306fc0dcb7bcdfaa8480f59869d95a3c12e8222567075"} Jul 06 19:13:23.179439 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:23.179387 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7" podStartSLOduration=1.661107135 podStartE2EDuration="6.179372415s" totalImagesPullingTime="4.51826528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:13:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:13:23.17885393 +0000 UTC m=+100.074743464" watchObservedRunningTime="2026-07-06 19:13:23.179372415 +0000 UTC m=+100.075261947" Jul 06 19:13:23.671881 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:23.671846 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:13:23.672056 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:13:23.671991 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:13:23.672056 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:13:23.672046 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls podName:bca4b412-168b-48a3-8251-28aea09c3427 nodeName:}" failed. No retries permitted until 2026-07-06 19:14:27.672030865 +0000 UTC m=+164.567920380 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls") pod "dns-default-62xmt" (UID: "bca4b412-168b-48a3-8251-28aea09c3427") : secret "dns-default-metrics-tls" not found Jul 06 19:13:24.069909 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:24.069827 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zxj86" Jul 06 19:13:26.155261 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:26.155224 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" event={"ID":"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de","Type":"ContainerStarted","Data":"0c29c01902745924f864293a86257141c6b417c534bf7d9458952a615ec42940"} Jul 06 19:13:26.155261 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:26.155268 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" event={"ID":"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de","Type":"ContainerStarted","Data":"fedd9fdf4490f9231798229357338687cc589395189b9a10edb400c9991d1583"} Jul 06 19:13:26.155697 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:26.155279 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" event={"ID":"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de","Type":"ContainerStarted","Data":"daf28a61254ad974f4a8f6a4cbba465ca956835060dce630c47b19cdde7d416f"} Jul 06 19:13:26.185141 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:26.185095 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" podStartSLOduration=1.182259366 podStartE2EDuration="8.185079805s" totalImagesPullingTime="7.002820439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:13:26.183988496 +0000 UTC m=+103.079878028" watchObservedRunningTime="2026-07-06 19:13:26.185079805 +0000 UTC m=+103.080969336" Jul 06 19:13:53.385461 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:13:53.385408 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs\") pod \"network-metrics-daemon-2pdkn\" (UID: \"2db677cb-ce90-46dd-b292-3d8c63da4fbb\") " pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:13:53.385995 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:13:53.385560 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 19:13:53.385995 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:13:53.385628 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs podName:2db677cb-ce90-46dd-b292-3d8c63da4fbb nodeName:}" failed. No retries permitted until 2026-07-06 19:15:55.385610586 +0000 UTC m=+252.281500096 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs") pod "network-metrics-daemon-2pdkn" (UID: "2db677cb-ce90-46dd-b292-3d8c63da4fbb") : secret "metrics-daemon-secret" not found Jul 06 19:14:22.841540 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:14:22.841480 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-62xmt" podUID="bca4b412-168b-48a3-8251-28aea09c3427" Jul 06 19:14:23.314198 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:14:23.314169 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-62xmt" Jul 06 19:14:24.776922 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:14:24.776873 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-2pdkn" podUID="2db677cb-ce90-46dd-b292-3d8c63da4fbb" Jul 06 19:14:27.739250 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:14:27.739205 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:14:27.739628 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:14:27.739363 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:14:27.739628 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:14:27.739436 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls podName:bca4b412-168b-48a3-8251-28aea09c3427 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:29.739417982 +0000 UTC m=+286.635307509 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls") pod "dns-default-62xmt" (UID: "bca4b412-168b-48a3-8251-28aea09c3427") : secret "dns-default-metrics-tls" not found Jul 06 19:14:38.757794 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:14:38.757692 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:15:23.483892 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:15:23.483858 2566 generic.go:356] "Generic (PLEG): container finished" podID="29fdffb8-755c-420f-91cb-eee58f087733" containerID="04d7a85c395c802cccc306fc0dcb7bcdfaa8480f59869d95a3c12e8222567075" exitCode=255 Jul 06 19:15:23.484374 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:15:23.483900 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7" event={"ID":"29fdffb8-755c-420f-91cb-eee58f087733","Type":"ContainerDied","Data":"04d7a85c395c802cccc306fc0dcb7bcdfaa8480f59869d95a3c12e8222567075"} Jul 06 19:15:23.484374 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:15:23.484143 2566 scope.go:117] "RemoveContainer" containerID="04d7a85c395c802cccc306fc0dcb7bcdfaa8480f59869d95a3c12e8222567075" Jul 06 19:15:24.488373 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:15:24.488336 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fbcfbc478-fcnh7" event={"ID":"29fdffb8-755c-420f-91cb-eee58f087733","Type":"ContainerStarted","Data":"f786884dbf5073078b06d26aba4902a910c9e7d83789fa9a28000f8ea701eaa6"} Jul 06 19:15:30.643910 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:15:30.643873 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:15:44.284218 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:15:44.284177 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:15:55.429868 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:15:55.429830 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs\") pod \"network-metrics-daemon-2pdkn\" (UID: \"2db677cb-ce90-46dd-b292-3d8c63da4fbb\") " pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:15:55.430282 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:15:55.429996 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 19:15:55.430282 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:15:55.430078 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs podName:2db677cb-ce90-46dd-b292-3d8c63da4fbb nodeName:}" failed. No retries permitted until 2026-07-06 19:17:57.430061277 +0000 UTC m=+374.325950787 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs") pod "network-metrics-daemon-2pdkn" (UID: "2db677cb-ce90-46dd-b292-3d8c63da4fbb") : secret "metrics-daemon-secret" not found Jul 06 19:15:56.922082 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:15:56.922043 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:15:57.024245 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:15:57.024211 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:15:58.882115 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:15:58.882075 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:15:58.982627 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:15:58.982597 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:02.061466 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.061429 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:02.122259 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.122223 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:02.303175 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.303143 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:02.451247 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.451215 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-78fpq"] Jul 06 19:16:02.454278 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.454255 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:02.456574 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.456552 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 19:16:02.457570 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.457551 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 19:16:02.457680 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.457611 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 19:16:02.457752 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.457680 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 19:16:02.457752 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.457709 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2n6g9\"" Jul 06 19:16:02.464061 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.464039 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-78fpq"] Jul 06 19:16:02.490962 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.490929 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:02.580467 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.580428 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82cca416-5961-4dae-ac5b-87f05514dea2-crio-socket\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:02.580631 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.580479 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82cca416-5961-4dae-ac5b-87f05514dea2-data-volume\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:02.580631 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.580560 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82cca416-5961-4dae-ac5b-87f05514dea2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:02.580631 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.580606 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:02.580631 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.580627 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb54h\" (UniqueName: \"kubernetes.io/projected/82cca416-5961-4dae-ac5b-87f05514dea2-kube-api-access-bb54h\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:02.681620 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.681575 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82cca416-5961-4dae-ac5b-87f05514dea2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:02.681851 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.681640 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:02.681851 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.681667 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb54h\" (UniqueName: \"kubernetes.io/projected/82cca416-5961-4dae-ac5b-87f05514dea2-kube-api-access-bb54h\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:02.681851 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.681707 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82cca416-5961-4dae-ac5b-87f05514dea2-crio-socket\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:02.681851 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:02.681711 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:16:02.681851 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.681771 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82cca416-5961-4dae-ac5b-87f05514dea2-data-volume\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:02.681851 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:02.681818 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82cca416-5961-4dae-ac5b-87f05514dea2-kube-rbac-proxy-cm podName:82cca416-5961-4dae-ac5b-87f05514dea2 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:03.181797924 +0000 UTC m=+260.077687453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82cca416-5961-4dae-ac5b-87f05514dea2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-78fpq" (UID: "82cca416-5961-4dae-ac5b-87f05514dea2") : configmap "kube-rbac-proxy" not found Jul 06 19:16:02.681851 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:02.681831 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:16:02.682244 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:02.681878 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls podName:82cca416-5961-4dae-ac5b-87f05514dea2 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:03.181865429 +0000 UTC m=+260.077754939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-78fpq" (UID: "82cca416-5961-4dae-ac5b-87f05514dea2") : secret "insights-runtime-extractor-tls" not found Jul 06 19:16:02.682244 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.681881 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82cca416-5961-4dae-ac5b-87f05514dea2-crio-socket\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:02.682244 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.682110 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82cca416-5961-4dae-ac5b-87f05514dea2-data-volume\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:02.689933 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.689911 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb54h\" (UniqueName: \"kubernetes.io/projected/82cca416-5961-4dae-ac5b-87f05514dea2-kube-api-access-bb54h\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:02.765636 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.765537 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:02.866420 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:02.866384 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:03.186459 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:03.186426 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82cca416-5961-4dae-ac5b-87f05514dea2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:03.186860 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:03.186471 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:03.186860 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:03.186546 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:16:03.186860 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:03.186626 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82cca416-5961-4dae-ac5b-87f05514dea2-kube-rbac-proxy-cm podName:82cca416-5961-4dae-ac5b-87f05514dea2 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:04.186608748 +0000 UTC m=+261.082498257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82cca416-5961-4dae-ac5b-87f05514dea2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-78fpq" (UID: "82cca416-5961-4dae-ac5b-87f05514dea2") : configmap "kube-rbac-proxy" not found Jul 06 19:16:03.186860 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:03.186556 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:16:03.186860 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:03.186672 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls podName:82cca416-5961-4dae-ac5b-87f05514dea2 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:04.186660992 +0000 UTC m=+261.082550502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-78fpq" (UID: "82cca416-5961-4dae-ac5b-87f05514dea2") : secret "insights-runtime-extractor-tls" not found Jul 06 19:16:03.577203 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:03.577122 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:03.675466 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:03.675431 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:03.777348 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:03.777309 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:04.193872 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.193833 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:04.194258 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.193925 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82cca416-5961-4dae-ac5b-87f05514dea2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:04.194258 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:04.193992 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:16:04.194258 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:04.194065 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls podName:82cca416-5961-4dae-ac5b-87f05514dea2 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:06.194048922 +0000 UTC m=+263.089938436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-78fpq" (UID: "82cca416-5961-4dae-ac5b-87f05514dea2") : secret "insights-runtime-extractor-tls" not found Jul 06 19:16:04.194258 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:04.193992 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:16:04.194258 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:04.194113 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82cca416-5961-4dae-ac5b-87f05514dea2-kube-rbac-proxy-cm podName:82cca416-5961-4dae-ac5b-87f05514dea2 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:06.194102514 +0000 UTC m=+263.089992026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82cca416-5961-4dae-ac5b-87f05514dea2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-78fpq" (UID: "82cca416-5961-4dae-ac5b-87f05514dea2") : configmap "kube-rbac-proxy" not found Jul 06 19:16:04.360193 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.360162 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-5x9j2"] Jul 06 19:16:04.363011 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.362992 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-5x9j2" Jul 06 19:16:04.365420 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.365400 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 06 19:16:04.366454 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.366432 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cpvdc\"" Jul 06 19:16:04.366454 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.366442 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 06 19:16:04.366454 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.366456 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 06 19:16:04.366669 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.366433 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 06 19:16:04.370501 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.370482 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-5x9j2"] Jul 06 19:16:04.473262 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.473180 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:04.496573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.496540 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq5p9\" (UniqueName: \"kubernetes.io/projected/03ccd006-8a63-4ce2-860c-e2920c2b3457-kube-api-access-cq5p9\") pod \"service-ca-758f599c5-5x9j2\" (UID: \"03ccd006-8a63-4ce2-860c-e2920c2b3457\") " pod="openshift-service-ca/service-ca-758f599c5-5x9j2" Jul 06 19:16:04.496683 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.496663 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/03ccd006-8a63-4ce2-860c-e2920c2b3457-signing-cabundle\") pod \"service-ca-758f599c5-5x9j2\" (UID: \"03ccd006-8a63-4ce2-860c-e2920c2b3457\") " pod="openshift-service-ca/service-ca-758f599c5-5x9j2" Jul 06 19:16:04.496764 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.496694 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/03ccd006-8a63-4ce2-860c-e2920c2b3457-signing-key\") pod \"service-ca-758f599c5-5x9j2\" (UID: \"03ccd006-8a63-4ce2-860c-e2920c2b3457\") " pod="openshift-service-ca/service-ca-758f599c5-5x9j2" Jul 06 19:16:04.574790 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.574753 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:04.597838 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.597810 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/03ccd006-8a63-4ce2-860c-e2920c2b3457-signing-cabundle\") pod \"service-ca-758f599c5-5x9j2\" (UID: \"03ccd006-8a63-4ce2-860c-e2920c2b3457\") " pod="openshift-service-ca/service-ca-758f599c5-5x9j2" Jul 06 19:16:04.597975 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.597848 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/03ccd006-8a63-4ce2-860c-e2920c2b3457-signing-key\") pod \"service-ca-758f599c5-5x9j2\" (UID: \"03ccd006-8a63-4ce2-860c-e2920c2b3457\") " pod="openshift-service-ca/service-ca-758f599c5-5x9j2" Jul 06 19:16:04.597975 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.597891 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq5p9\" (UniqueName: \"kubernetes.io/projected/03ccd006-8a63-4ce2-860c-e2920c2b3457-kube-api-access-cq5p9\") pod \"service-ca-758f599c5-5x9j2\" (UID: \"03ccd006-8a63-4ce2-860c-e2920c2b3457\") " pod="openshift-service-ca/service-ca-758f599c5-5x9j2" Jul 06 19:16:04.598454 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.598428 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/03ccd006-8a63-4ce2-860c-e2920c2b3457-signing-cabundle\") pod \"service-ca-758f599c5-5x9j2\" (UID: \"03ccd006-8a63-4ce2-860c-e2920c2b3457\") " pod="openshift-service-ca/service-ca-758f599c5-5x9j2" Jul 06 19:16:04.600304 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.600285 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/03ccd006-8a63-4ce2-860c-e2920c2b3457-signing-key\") pod \"service-ca-758f599c5-5x9j2\" (UID: \"03ccd006-8a63-4ce2-860c-e2920c2b3457\") " pod="openshift-service-ca/service-ca-758f599c5-5x9j2" Jul 06 19:16:04.606798 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.606778 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq5p9\" (UniqueName: \"kubernetes.io/projected/03ccd006-8a63-4ce2-860c-e2920c2b3457-kube-api-access-cq5p9\") pod \"service-ca-758f599c5-5x9j2\" (UID: \"03ccd006-8a63-4ce2-860c-e2920c2b3457\") " pod="openshift-service-ca/service-ca-758f599c5-5x9j2" Jul 06 19:16:04.641351 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.641325 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zcnp6_a751eaf8-99c8-4e12-92a7-e135a18167a9/dns-node-resolver/0.log" Jul 06 19:16:04.672501 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.672474 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-5x9j2" Jul 06 19:16:04.800517 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.800484 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-5x9j2"] Jul 06 19:16:04.804383 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:16:04.804357 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ccd006_8a63_4ce2_860c_e2920c2b3457.slice/crio-4f39e6f611ffd22297d36a944c77ffcf07bf12a4c826d59452bef7bcbf165b29 WatchSource:0}: Error finding container 4f39e6f611ffd22297d36a944c77ffcf07bf12a4c826d59452bef7bcbf165b29: Status 404 returned error can't find the container with id 4f39e6f611ffd22297d36a944c77ffcf07bf12a4c826d59452bef7bcbf165b29 Jul 06 19:16:04.950010 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:04.949935 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:05.241693 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:05.241665 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-454zs_5f592f36-c320-4ddb-abca-8bc3f6586ae0/node-ca/0.log" Jul 06 19:16:05.481132 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:05.481081 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:05.604608 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:05.604520 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-5x9j2" event={"ID":"03ccd006-8a63-4ce2-860c-e2920c2b3457","Type":"ContainerStarted","Data":"4f39e6f611ffd22297d36a944c77ffcf07bf12a4c826d59452bef7bcbf165b29"} Jul 06 19:16:06.213289 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:06.213249 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82cca416-5961-4dae-ac5b-87f05514dea2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:06.213472 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:06.213301 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:06.213472 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:06.213386 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:16:06.213472 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:06.213407 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:16:06.213472 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:06.213453 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82cca416-5961-4dae-ac5b-87f05514dea2-kube-rbac-proxy-cm podName:82cca416-5961-4dae-ac5b-87f05514dea2 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:10.213435102 +0000 UTC m=+267.109324612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82cca416-5961-4dae-ac5b-87f05514dea2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-78fpq" (UID: "82cca416-5961-4dae-ac5b-87f05514dea2") : configmap "kube-rbac-proxy" not found Jul 06 19:16:06.213472 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:06.213469 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls podName:82cca416-5961-4dae-ac5b-87f05514dea2 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:10.21346284 +0000 UTC m=+267.109352350 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-78fpq" (UID: "82cca416-5961-4dae-ac5b-87f05514dea2") : secret "insights-runtime-extractor-tls" not found Jul 06 19:16:06.441343 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:06.441314 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mk4bf_3831b65a-0a72-4c06-b6d4-b3bff61a5a32/serve-healthcheck-canary/0.log" Jul 06 19:16:08.203690 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:08.203653 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:08.616385 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:08.616304 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-5x9j2" event={"ID":"03ccd006-8a63-4ce2-860c-e2920c2b3457","Type":"ContainerStarted","Data":"623148eb7ae236cadb49c022ad2494f9fc05dc099c14f61cd921eeaef9d158c8"} Jul 06 19:16:08.632689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:08.632643 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-5x9j2" podStartSLOduration=1.86707544 podStartE2EDuration="4.632625296s" totalImagesPullingTime="2.765549856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:08.631818209 +0000 UTC m=+265.527707742" watchObservedRunningTime="2026-07-06 19:16:08.632625296 +0000 UTC m=+265.528514828" Jul 06 19:16:08.811350 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:08.811310 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:08.952813 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:08.952777 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:09.057826 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:09.057796 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:09.158443 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:09.158392 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:09.259390 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:09.259294 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:09.358302 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:09.358266 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:09.530980 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:09.530886 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:10.247617 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:10.247564 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82cca416-5961-4dae-ac5b-87f05514dea2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:10.247856 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:10.247651 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:10.247856 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:10.247705 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:16:10.247856 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:10.247791 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82cca416-5961-4dae-ac5b-87f05514dea2-kube-rbac-proxy-cm podName:82cca416-5961-4dae-ac5b-87f05514dea2 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:18.247776932 +0000 UTC m=+275.143666447 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82cca416-5961-4dae-ac5b-87f05514dea2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-78fpq" (UID: "82cca416-5961-4dae-ac5b-87f05514dea2") : configmap "kube-rbac-proxy" not found Jul 06 19:16:10.247856 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:10.247837 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:16:10.248035 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:10.247904 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls podName:82cca416-5961-4dae-ac5b-87f05514dea2 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:18.247886194 +0000 UTC m=+275.143775716 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-78fpq" (UID: "82cca416-5961-4dae-ac5b-87f05514dea2") : secret "insights-runtime-extractor-tls" not found Jul 06 19:16:10.511556 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:10.511465 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:10.916131 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:10.916102 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:11.103229 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:11.103193 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:11.325536 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:11.325449 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:11.510266 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:11.510234 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:12.507636 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:12.507595 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:12.580667 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:12.580632 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:12.707755 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:12.707708 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:13.038487 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:13.038446 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:13.139838 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:13.139809 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:13.306930 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:13.306840 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:13.703591 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:13.703553 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:14.307972 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:14.307937 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:14.908522 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:14.908486 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:16.543643 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:16.543607 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:16.652885 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:16.652850 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:16.915166 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:16.915138 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:17.106134 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:17.106095 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:17.313625 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:17.313525 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:17.708441 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:17.708396 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:18.306291 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:18.306252 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:18.309447 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:18.309429 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82cca416-5961-4dae-ac5b-87f05514dea2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:18.309554 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:18.309470 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:18.309668 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:18.309570 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:16:18.309668 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:18.309630 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls podName:82cca416-5961-4dae-ac5b-87f05514dea2 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:34.309616773 +0000 UTC m=+291.205506283 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-78fpq" (UID: "82cca416-5961-4dae-ac5b-87f05514dea2") : secret "insights-runtime-extractor-tls" not found Jul 06 19:16:18.310014 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:18.309994 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82cca416-5961-4dae-ac5b-87f05514dea2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:18.708069 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:18.708018 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:19.110762 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:19.110644 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:19.310092 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:19.310060 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:19.511858 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:19.511817 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:21.005448 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:21.005414 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:21.071065 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:21.071033 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:21.132037 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:21.132009 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:21.520836 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:21.520810 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:21.691827 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:21.691793 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:21.868360 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:21.868280 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:21.926036 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:21.926006 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:22.079383 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:22.079352 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:22.296089 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:22.296045 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:22.348910 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:22.348875 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:22.489116 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:22.489087 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:22.589668 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:22.589565 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:22.691764 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:22.691716 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:22.884414 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:22.884320 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:23.274257 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:23.274225 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:24.314092 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:24.314050 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:24.437570 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:24.437534 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:26.315318 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:26.315268 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-62xmt" podUID="bca4b412-168b-48a3-8251-28aea09c3427" Jul 06 19:16:26.670290 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:26.670262 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-62xmt" Jul 06 19:16:28.042521 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:28.042478 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:28.134804 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:28.134774 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:29.792813 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:29.792765 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:16:29.795134 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:29.795111 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bca4b412-168b-48a3-8251-28aea09c3427-metrics-tls\") pod \"dns-default-62xmt\" (UID: \"bca4b412-168b-48a3-8251-28aea09c3427\") " pod="openshift-dns/dns-default-62xmt" Jul 06 19:16:29.973190 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:29.973162 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c8xcq\"" Jul 06 19:16:29.981151 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:29.981135 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-62xmt" Jul 06 19:16:30.110981 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:30.110951 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-62xmt"] Jul 06 19:16:30.114035 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:16:30.114009 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca4b412_168b_48a3_8251_28aea09c3427.slice/crio-aef553029f7ec9c6f88cea127496e032f3ffaebebaf87921ed9d8ff22f94c5f0 WatchSource:0}: Error finding container aef553029f7ec9c6f88cea127496e032f3ffaebebaf87921ed9d8ff22f94c5f0: Status 404 returned error can't find the container with id aef553029f7ec9c6f88cea127496e032f3ffaebebaf87921ed9d8ff22f94c5f0 Jul 06 19:16:30.683198 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:30.683160 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-62xmt" event={"ID":"bca4b412-168b-48a3-8251-28aea09c3427","Type":"ContainerStarted","Data":"aef553029f7ec9c6f88cea127496e032f3ffaebebaf87921ed9d8ff22f94c5f0"} Jul 06 19:16:31.784129 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:31.784088 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:31.881515 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:31.881471 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:31.968354 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:31.968304 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:32.213651 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:32.213622 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:32.690406 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:32.690369 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-62xmt" event={"ID":"bca4b412-168b-48a3-8251-28aea09c3427","Type":"ContainerStarted","Data":"f9808e6f68bda8623f8213645033419e9398c8f341ff31fcee21b6daf3ec59ed"} Jul 06 19:16:32.690587 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:32.690431 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-62xmt" event={"ID":"bca4b412-168b-48a3-8251-28aea09c3427","Type":"ContainerStarted","Data":"9d965681295f1f3057ce0d93c425e5aba455aceecd0cad270a41167d3300b8c1"} Jul 06 19:16:32.690587 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:32.690486 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-62xmt" Jul 06 19:16:32.717986 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:32.717942 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-62xmt" podStartSLOduration=251.77148811 podStartE2EDuration="4m13.717925731s" totalImagesPullingTime="1.946437621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:32.717635328 +0000 UTC m=+289.613524861" watchObservedRunningTime="2026-07-06 19:16:32.717925731 +0000 UTC m=+289.613815257" Jul 06 19:16:34.327913 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:34.327872 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:34.330085 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:34.330066 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82cca416-5961-4dae-ac5b-87f05514dea2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-78fpq\" (UID: \"82cca416-5961-4dae-ac5b-87f05514dea2\") " pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:34.563471 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:34.563439 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-78fpq" Jul 06 19:16:34.686548 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:34.686431 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-78fpq"] Jul 06 19:16:34.689305 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:16:34.689276 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82cca416_5961_4dae_ac5b_87f05514dea2.slice/crio-c0659a68143c374e8f8048653f040fe58bf7947935e9b328a0eff4c97b2b8f61 WatchSource:0}: Error finding container c0659a68143c374e8f8048653f040fe58bf7947935e9b328a0eff4c97b2b8f61: Status 404 returned error can't find the container with id c0659a68143c374e8f8048653f040fe58bf7947935e9b328a0eff4c97b2b8f61 Jul 06 19:16:34.696296 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:34.696273 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-78fpq" event={"ID":"82cca416-5961-4dae-ac5b-87f05514dea2","Type":"ContainerStarted","Data":"c0659a68143c374e8f8048653f040fe58bf7947935e9b328a0eff4c97b2b8f61"} Jul 06 19:16:35.972930 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:35.972887 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:36.067257 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:36.067218 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:36.228096 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:36.228006 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:37.083626 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:37.083588 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:37.143352 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:37.143317 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:38.124046 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:38.124006 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:38.224048 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:38.224012 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:38.597842 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:38.597779 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:38.697901 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:38.697867 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:38.713302 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:38.713273 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-78fpq" event={"ID":"82cca416-5961-4dae-ac5b-87f05514dea2","Type":"ContainerStarted","Data":"c25d906be42618a6f9945798c623586c0a5c251bef5cc6a13c5c268e0971c344"} Jul 06 19:16:39.306098 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:39.306059 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:39.406299 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:39.406258 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:39.718405 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:39.718367 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-78fpq" event={"ID":"82cca416-5961-4dae-ac5b-87f05514dea2","Type":"ContainerStarted","Data":"d309d19b8f2588bdf9ef796445ef8efcd5108742c74bcc8af43751ccfe4b4b9b"} Jul 06 19:16:40.417310 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:40.417273 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:41.728807 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:41.728775 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-78fpq" event={"ID":"82cca416-5961-4dae-ac5b-87f05514dea2","Type":"ContainerStarted","Data":"14d42abbe5284d9f9c4275adb4624e38e884a1d447f96495abcc12b426c67373"} Jul 06 19:16:41.749927 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:41.749872 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-78fpq" podStartSLOduration=32.895488851 podStartE2EDuration="39.749857092s" totalImagesPullingTime="6.854368241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:16:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:41.748673825 +0000 UTC m=+298.644563359" watchObservedRunningTime="2026-07-06 19:16:41.749857092 +0000 UTC m=+298.645746623" Jul 06 19:16:41.758815 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:41.758778 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-2pdkn" podUID="2db677cb-ce90-46dd-b292-3d8c63da4fbb" Jul 06 19:16:42.150290 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.150258 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:42.355792 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.355757 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:42.458892 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.458860 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm"] Jul 06 19:16:42.460917 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.460901 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" Jul 06 19:16:42.465071 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.464884 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8kgwm\"" Jul 06 19:16:42.465071 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.464906 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 19:16:42.465071 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.464936 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 06 19:16:42.465071 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.464947 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 19:16:42.465071 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.464959 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 19:16:42.465399 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.465178 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 06 19:16:42.465399 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.465184 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 06 19:16:42.471340 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.471305 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:42.479687 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.479664 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm"] Jul 06 19:16:42.480848 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.480829 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2cpjz"] Jul 06 19:16:42.483245 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.483220 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.486084 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.486065 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-l478r\"" Jul 06 19:16:42.486211 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.486077 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 19:16:42.486324 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.486314 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 19:16:42.486444 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.486428 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 19:16:42.488484 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.488463 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce7fbf1a-70ea-478d-914f-ca46298998aa-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-9rhvm\" (UID: \"ce7fbf1a-70ea-478d-914f-ca46298998aa\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" Jul 06 19:16:42.488589 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.488491 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce7fbf1a-70ea-478d-914f-ca46298998aa-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-9rhvm\" (UID: \"ce7fbf1a-70ea-478d-914f-ca46298998aa\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" Jul 06 19:16:42.488589 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.488522 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ce7fbf1a-70ea-478d-914f-ca46298998aa-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-9rhvm\" (UID: \"ce7fbf1a-70ea-478d-914f-ca46298998aa\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" Jul 06 19:16:42.488704 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.488623 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqv9x\" (UniqueName: \"kubernetes.io/projected/ce7fbf1a-70ea-478d-914f-ca46298998aa-kube-api-access-wqv9x\") pod \"kube-state-metrics-7c9dbb8b7d-9rhvm\" (UID: \"ce7fbf1a-70ea-478d-914f-ca46298998aa\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" Jul 06 19:16:42.488704 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.488660 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce7fbf1a-70ea-478d-914f-ca46298998aa-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-9rhvm\" (UID: \"ce7fbf1a-70ea-478d-914f-ca46298998aa\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" Jul 06 19:16:42.488827 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.488702 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ce7fbf1a-70ea-478d-914f-ca46298998aa-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-9rhvm\" (UID: \"ce7fbf1a-70ea-478d-914f-ca46298998aa\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" Jul 06 19:16:42.565252 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.565221 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:42.589298 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.589263 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ce7fbf1a-70ea-478d-914f-ca46298998aa-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-9rhvm\" (UID: \"ce7fbf1a-70ea-478d-914f-ca46298998aa\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" Jul 06 19:16:42.589477 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.589310 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqv9x\" (UniqueName: \"kubernetes.io/projected/ce7fbf1a-70ea-478d-914f-ca46298998aa-kube-api-access-wqv9x\") pod \"kube-state-metrics-7c9dbb8b7d-9rhvm\" (UID: \"ce7fbf1a-70ea-478d-914f-ca46298998aa\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" Jul 06 19:16:42.589477 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.589341 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f96f9815-cbfa-4b08-b3e9-52b7368232c6-root\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.589477 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.589363 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f96f9815-cbfa-4b08-b3e9-52b7368232c6-node-exporter-tls\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.589477 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.589386 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce7fbf1a-70ea-478d-914f-ca46298998aa-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-9rhvm\" (UID: \"ce7fbf1a-70ea-478d-914f-ca46298998aa\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" Jul 06 19:16:42.589477 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.589406 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f96f9815-cbfa-4b08-b3e9-52b7368232c6-node-exporter-textfile\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.589477 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.589428 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f96f9815-cbfa-4b08-b3e9-52b7368232c6-metrics-client-ca\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.589477 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.589465 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ce7fbf1a-70ea-478d-914f-ca46298998aa-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-9rhvm\" (UID: \"ce7fbf1a-70ea-478d-914f-ca46298998aa\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" Jul 06 19:16:42.589831 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.589514 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h6qs\" (UniqueName: \"kubernetes.io/projected/f96f9815-cbfa-4b08-b3e9-52b7368232c6-kube-api-access-5h6qs\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.589831 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.589579 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f96f9815-cbfa-4b08-b3e9-52b7368232c6-sys\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.589831 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.589608 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f96f9815-cbfa-4b08-b3e9-52b7368232c6-node-exporter-wtmp\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.589831 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.589657 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ce7fbf1a-70ea-478d-914f-ca46298998aa-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-9rhvm\" (UID: \"ce7fbf1a-70ea-478d-914f-ca46298998aa\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" Jul 06 19:16:42.589831 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.589659 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f96f9815-cbfa-4b08-b3e9-52b7368232c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.589831 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.589763 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f96f9815-cbfa-4b08-b3e9-52b7368232c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.589831 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.589792 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce7fbf1a-70ea-478d-914f-ca46298998aa-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-9rhvm\" (UID: \"ce7fbf1a-70ea-478d-914f-ca46298998aa\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" Jul 06 19:16:42.589831 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.589812 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce7fbf1a-70ea-478d-914f-ca46298998aa-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-9rhvm\" (UID: \"ce7fbf1a-70ea-478d-914f-ca46298998aa\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" Jul 06 19:16:42.590192 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.590171 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce7fbf1a-70ea-478d-914f-ca46298998aa-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-9rhvm\" (UID: \"ce7fbf1a-70ea-478d-914f-ca46298998aa\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" Jul 06 19:16:42.590248 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.590171 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ce7fbf1a-70ea-478d-914f-ca46298998aa-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-9rhvm\" (UID: \"ce7fbf1a-70ea-478d-914f-ca46298998aa\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" Jul 06 19:16:42.592255 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.592230 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce7fbf1a-70ea-478d-914f-ca46298998aa-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-9rhvm\" (UID: \"ce7fbf1a-70ea-478d-914f-ca46298998aa\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" Jul 06 19:16:42.592366 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.592274 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce7fbf1a-70ea-478d-914f-ca46298998aa-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-9rhvm\" (UID: \"ce7fbf1a-70ea-478d-914f-ca46298998aa\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" Jul 06 19:16:42.601571 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.601546 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqv9x\" (UniqueName: \"kubernetes.io/projected/ce7fbf1a-70ea-478d-914f-ca46298998aa-kube-api-access-wqv9x\") pod \"kube-state-metrics-7c9dbb8b7d-9rhvm\" (UID: \"ce7fbf1a-70ea-478d-914f-ca46298998aa\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" Jul 06 19:16:42.690770 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.690710 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f96f9815-cbfa-4b08-b3e9-52b7368232c6-root\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.690770 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.690768 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f96f9815-cbfa-4b08-b3e9-52b7368232c6-node-exporter-tls\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.690993 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.690828 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f96f9815-cbfa-4b08-b3e9-52b7368232c6-root\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.690993 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.690890 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f96f9815-cbfa-4b08-b3e9-52b7368232c6-node-exporter-textfile\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.690993 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.690932 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f96f9815-cbfa-4b08-b3e9-52b7368232c6-metrics-client-ca\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.690993 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.690968 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h6qs\" (UniqueName: \"kubernetes.io/projected/f96f9815-cbfa-4b08-b3e9-52b7368232c6-kube-api-access-5h6qs\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.691195 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.690998 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f96f9815-cbfa-4b08-b3e9-52b7368232c6-sys\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.691195 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.691030 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f96f9815-cbfa-4b08-b3e9-52b7368232c6-node-exporter-wtmp\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.691195 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.691068 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f96f9815-cbfa-4b08-b3e9-52b7368232c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.691195 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.691105 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f96f9815-cbfa-4b08-b3e9-52b7368232c6-sys\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.691195 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.691127 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f96f9815-cbfa-4b08-b3e9-52b7368232c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.691436 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.691200 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f96f9815-cbfa-4b08-b3e9-52b7368232c6-node-exporter-wtmp\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.691436 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.691264 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f96f9815-cbfa-4b08-b3e9-52b7368232c6-node-exporter-textfile\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.691646 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.691622 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f96f9815-cbfa-4b08-b3e9-52b7368232c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.691750 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.691705 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f96f9815-cbfa-4b08-b3e9-52b7368232c6-metrics-client-ca\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.693098 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.693068 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f96f9815-cbfa-4b08-b3e9-52b7368232c6-node-exporter-tls\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.693489 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.693467 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f96f9815-cbfa-4b08-b3e9-52b7368232c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.694782 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.694760 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-62xmt" Jul 06 19:16:42.702901 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.702872 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h6qs\" (UniqueName: \"kubernetes.io/projected/f96f9815-cbfa-4b08-b3e9-52b7368232c6-kube-api-access-5h6qs\") pod \"node-exporter-2cpjz\" (UID: \"f96f9815-cbfa-4b08-b3e9-52b7368232c6\") " pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.769527 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.769393 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:42.770237 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.770210 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" Jul 06 19:16:42.792347 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.792320 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2cpjz" Jul 06 19:16:42.807497 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:16:42.807472 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96f9815_cbfa_4b08_b3e9_52b7368232c6.slice/crio-5c0215704418495c1c1e8a77cb0dd332793f308a379d4c87daca7906cb1a739a WatchSource:0}: Error finding container 5c0215704418495c1c1e8a77cb0dd332793f308a379d4c87daca7906cb1a739a: Status 404 returned error can't find the container with id 5c0215704418495c1c1e8a77cb0dd332793f308a379d4c87daca7906cb1a739a Jul 06 19:16:42.908383 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:42.908351 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm"] Jul 06 19:16:42.913083 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:16:42.913052 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7fbf1a_70ea_478d_914f_ca46298998aa.slice/crio-18ff137fc6884def519e824652679b7588fb9b43d930d3327df4e31de4e70e9a WatchSource:0}: Error finding container 18ff137fc6884def519e824652679b7588fb9b43d930d3327df4e31de4e70e9a: Status 404 returned error can't find the container with id 18ff137fc6884def519e824652679b7588fb9b43d930d3327df4e31de4e70e9a Jul 06 19:16:43.095943 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:43.095855 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:43.365349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:43.365261 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:43.461625 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:43.461586 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:43.528780 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:43.528747 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:16:43.534171 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:43.534141 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:16:43.535274 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:43.535252 2566 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 19:16:43.537117 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:43.537084 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:43.738251 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:43.738072 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2cpjz" event={"ID":"f96f9815-cbfa-4b08-b3e9-52b7368232c6","Type":"ContainerStarted","Data":"5c0215704418495c1c1e8a77cb0dd332793f308a379d4c87daca7906cb1a739a"} Jul 06 19:16:43.756716 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:43.739711 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" event={"ID":"ce7fbf1a-70ea-478d-914f-ca46298998aa","Type":"ContainerStarted","Data":"18ff137fc6884def519e824652679b7588fb9b43d930d3327df4e31de4e70e9a"} Jul 06 19:16:43.766390 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:43.766354 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:43.863806 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:43.863774 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:44.829483 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:44.829386 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:45.026480 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:45.026423 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:45.125492 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:45.125453 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:45.423692 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:45.423640 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:45.513402 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:45.513367 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:45.650198 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:45.650167 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:45.748973 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:45.748888 2566 generic.go:356] "Generic (PLEG): container finished" podID="f96f9815-cbfa-4b08-b3e9-52b7368232c6" containerID="b265da6132649e73fce716bbf7d223e14395f54048103f884c613624e77cc416" exitCode=0 Jul 06 19:16:45.749131 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:45.748971 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2cpjz" event={"ID":"f96f9815-cbfa-4b08-b3e9-52b7368232c6","Type":"ContainerDied","Data":"b265da6132649e73fce716bbf7d223e14395f54048103f884c613624e77cc416"} Jul 06 19:16:45.751212 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:45.751181 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" event={"ID":"ce7fbf1a-70ea-478d-914f-ca46298998aa","Type":"ContainerStarted","Data":"d00f302387b15b667ecd0f0540cfb3db3881930eb154f585996aa8e6575e5c09"} Jul 06 19:16:45.751343 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:45.751232 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" event={"ID":"ce7fbf1a-70ea-478d-914f-ca46298998aa","Type":"ContainerStarted","Data":"b0a6225797909418e8ca8123a38b18c6926cc73b01e9963d7e31c18894265e7f"} Jul 06 19:16:45.751343 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:45.751266 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" event={"ID":"ce7fbf1a-70ea-478d-914f-ca46298998aa","Type":"ContainerStarted","Data":"2a8b265dd413fe2ae509f4839f97f801296dec038b1f0370363351a488a44155"} Jul 06 19:16:45.790457 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:45.790385 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-9rhvm" podStartSLOduration=1.980178357 podStartE2EDuration="3.790369867s" totalImagesPullingTime="1.81019151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:45.789747442 +0000 UTC m=+302.685636974" watchObservedRunningTime="2026-07-06 19:16:45.790369867 +0000 UTC m=+302.686259398" Jul 06 19:16:45.888809 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:45.888774 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:46.756993 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:46.756953 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2cpjz" event={"ID":"f96f9815-cbfa-4b08-b3e9-52b7368232c6","Type":"ContainerStarted","Data":"cbf7b11b249b18e387a4c8bbc4309b8b8e39e60e88a9edef86e67ef9b6a7d6d8"} Jul 06 19:16:46.757361 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:46.757016 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2cpjz" event={"ID":"f96f9815-cbfa-4b08-b3e9-52b7368232c6","Type":"ContainerStarted","Data":"c0e853421321137485a2d8e7e6cdf422917c8d6f29a122a2dfd9ee7f5bc56a49"} Jul 06 19:16:47.174705 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:47.174668 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:47.180742 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:47.180683 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2cpjz" podStartSLOduration=3.179259279 podStartE2EDuration="5.180665429s" totalImagesPullingTime="1.913934589s" totalInitContainerRuntime="87.471561ms" isStatefulPod=true podCreationTimestamp="2026-07-06 19:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:46.781742637 +0000 UTC m=+303.677632177" watchObservedRunningTime="2026-07-06 19:16:47.180665429 +0000 UTC m=+304.076554961" Jul 06 19:16:47.181556 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:47.181540 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-9ghq4"] Jul 06 19:16:47.184806 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:47.184789 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9ghq4" Jul 06 19:16:47.187241 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:47.187222 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 06 19:16:47.187323 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:47.187252 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-xzrsc\"" Jul 06 19:16:47.195700 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:47.195676 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-9ghq4"] Jul 06 19:16:47.231482 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:47.231454 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mt89\" (UniqueName: \"kubernetes.io/projected/9b43d782-fd1c-4822-9944-dca222c4887a-kube-api-access-8mt89\") pod \"monitoring-plugin-d5dcff7cf-9ghq4\" (UID: \"9b43d782-fd1c-4822-9944-dca222c4887a\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9ghq4" Jul 06 19:16:47.231630 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:47.231518 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b43d782-fd1c-4822-9944-dca222c4887a-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-9ghq4\" (UID: \"9b43d782-fd1c-4822-9944-dca222c4887a\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9ghq4" Jul 06 19:16:47.252162 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:47.252137 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:47.332563 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:47.332530 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mt89\" (UniqueName: \"kubernetes.io/projected/9b43d782-fd1c-4822-9944-dca222c4887a-kube-api-access-8mt89\") pod \"monitoring-plugin-d5dcff7cf-9ghq4\" (UID: \"9b43d782-fd1c-4822-9944-dca222c4887a\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9ghq4" Jul 06 19:16:47.332808 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:47.332610 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b43d782-fd1c-4822-9944-dca222c4887a-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-9ghq4\" (UID: \"9b43d782-fd1c-4822-9944-dca222c4887a\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9ghq4" Jul 06 19:16:47.332808 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:47.332711 2566 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 06 19:16:47.332808 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:16:47.332805 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b43d782-fd1c-4822-9944-dca222c4887a-monitoring-plugin-cert podName:9b43d782-fd1c-4822-9944-dca222c4887a nodeName:}" failed. No retries permitted until 2026-07-06 19:16:47.832786952 +0000 UTC m=+304.728676465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/9b43d782-fd1c-4822-9944-dca222c4887a-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-9ghq4" (UID: "9b43d782-fd1c-4822-9944-dca222c4887a") : secret "monitoring-plugin-cert" not found Jul 06 19:16:47.342141 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:47.342117 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mt89\" (UniqueName: \"kubernetes.io/projected/9b43d782-fd1c-4822-9944-dca222c4887a-kube-api-access-8mt89\") pod \"monitoring-plugin-d5dcff7cf-9ghq4\" (UID: \"9b43d782-fd1c-4822-9944-dca222c4887a\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9ghq4" Jul 06 19:16:47.490461 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:47.490371 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:47.688350 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:47.688312 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:47.791201 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:47.791117 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:47.836381 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:47.836353 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b43d782-fd1c-4822-9944-dca222c4887a-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-9ghq4\" (UID: \"9b43d782-fd1c-4822-9944-dca222c4887a\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9ghq4" Jul 06 19:16:47.838814 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:47.838792 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b43d782-fd1c-4822-9944-dca222c4887a-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-9ghq4\" (UID: \"9b43d782-fd1c-4822-9944-dca222c4887a\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9ghq4" Jul 06 19:16:48.004308 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:48.004271 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:48.094005 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:48.093905 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9ghq4" Jul 06 19:16:48.106460 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:48.106401 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:48.218132 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:48.218102 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-9ghq4"] Jul 06 19:16:48.223752 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:16:48.223699 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b43d782_fd1c_4822_9944_dca222c4887a.slice/crio-a6fee8ec02477a7e437639bec40c784275c9fe12a1e9cf1304efe24d63614925 WatchSource:0}: Error finding container a6fee8ec02477a7e437639bec40c784275c9fe12a1e9cf1304efe24d63614925: Status 404 returned error can't find the container with id a6fee8ec02477a7e437639bec40c784275c9fe12a1e9cf1304efe24d63614925 Jul 06 19:16:48.226306 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:48.226290 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:16:48.760670 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:48.760638 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:48.763990 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:48.763965 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9ghq4" event={"ID":"9b43d782-fd1c-4822-9944-dca222c4887a","Type":"ContainerStarted","Data":"a6fee8ec02477a7e437639bec40c784275c9fe12a1e9cf1304efe24d63614925"} Jul 06 19:16:49.114583 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:49.114457 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:49.214170 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:49.214135 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:50.167670 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:50.167641 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:50.263881 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:50.263838 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:50.336922 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:50.336888 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:50.772486 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:50.772453 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9ghq4" event={"ID":"9b43d782-fd1c-4822-9944-dca222c4887a","Type":"ContainerStarted","Data":"64be6e4c60065b2e05085ddc8f6ed8fce48473c31ec96a3f0e4a1e982d85e299"} Jul 06 19:16:50.772699 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:50.772625 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9ghq4" Jul 06 19:16:50.777358 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:50.777336 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9ghq4" Jul 06 19:16:50.790099 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:50.790053 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9ghq4" podStartSLOduration=1.881685766 podStartE2EDuration="3.790039147s" totalImagesPullingTime="1.908353381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:50.788802541 +0000 UTC m=+307.684692075" watchObservedRunningTime="2026-07-06 19:16:50.790039147 +0000 UTC m=+307.685928679" Jul 06 19:16:51.368144 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:51.368117 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:51.441600 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:51.441571 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:52.482907 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:52.482855 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:52.580379 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:52.580343 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:55.758080 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:55.758042 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:16:56.999220 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:56.999180 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:57.062803 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:57.062760 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:59.296512 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:59.296479 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:59.397816 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:59.397784 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:59.457465 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:16:59.457432 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:17:00.137948 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:00.137915 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:17:00.207298 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:00.207267 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:17:02.105920 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:02.105885 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:17:07.479911 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:07.479880 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:17:07.577144 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:07.577113 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:17:08.392763 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:08.392698 2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" podUID="8b1f7b1f-070b-4c96-928e-ec47bf2dc8de" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 06 19:17:18.392011 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:18.391969 2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" podUID="8b1f7b1f-070b-4c96-928e-ec47bf2dc8de" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 06 19:17:22.457401 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:22.457363 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:17:22.513101 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:22.513043 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:17:28.392111 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:28.392073 2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" podUID="8b1f7b1f-070b-4c96-928e-ec47bf2dc8de" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 06 19:17:28.392513 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:28.392148 2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" Jul 06 19:17:28.392621 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:28.392581 2566 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"0c29c01902745924f864293a86257141c6b417c534bf7d9458952a615ec42940"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 06 19:17:28.392661 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:28.392640 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" podUID="8b1f7b1f-070b-4c96-928e-ec47bf2dc8de" containerName="service-proxy" containerID="cri-o://0c29c01902745924f864293a86257141c6b417c534bf7d9458952a615ec42940" gracePeriod=30 Jul 06 19:17:28.895016 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:28.894981 2566 generic.go:356] "Generic (PLEG): container finished" podID="8b1f7b1f-070b-4c96-928e-ec47bf2dc8de" containerID="0c29c01902745924f864293a86257141c6b417c534bf7d9458952a615ec42940" exitCode=2 Jul 06 19:17:28.895221 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:28.895061 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" event={"ID":"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de","Type":"ContainerDied","Data":"0c29c01902745924f864293a86257141c6b417c534bf7d9458952a615ec42940"} Jul 06 19:17:28.895221 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:28.895113 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-864494d4c7-t4mfq" event={"ID":"8b1f7b1f-070b-4c96-928e-ec47bf2dc8de","Type":"ContainerStarted","Data":"8056444df6b3b0e06dc89d56a5813a5557d778d9a688749416c0ccc647fbaaf8"} Jul 06 19:17:46.457372 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:46.457271 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:17:46.558139 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:46.558104 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:17:55.499845 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:55.499811 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:17:55.557064 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:55.557017 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:17:55.627590 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:55.627558 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:17:57.467432 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:57.467392 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs\") pod \"network-metrics-daemon-2pdkn\" (UID: \"2db677cb-ce90-46dd-b292-3d8c63da4fbb\") " pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:17:57.469913 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:57.469888 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db677cb-ce90-46dd-b292-3d8c63da4fbb-metrics-certs\") pod \"network-metrics-daemon-2pdkn\" (UID: \"2db677cb-ce90-46dd-b292-3d8c63da4fbb\") " pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:17:57.562301 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:57.562269 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r97mf\"" Jul 06 19:17:57.569432 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:57.569395 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pdkn" Jul 06 19:17:57.714948 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:57.714915 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2pdkn"] Jul 06 19:17:57.718067 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:17:57.717994 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db677cb_ce90_46dd_b292_3d8c63da4fbb.slice/crio-74845fc9172e2961131a72386fb3a692796a0c7135ae0e83751df003893d6f28 WatchSource:0}: Error finding container 74845fc9172e2961131a72386fb3a692796a0c7135ae0e83751df003893d6f28: Status 404 returned error can't find the container with id 74845fc9172e2961131a72386fb3a692796a0c7135ae0e83751df003893d6f28 Jul 06 19:17:57.991376 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:57.991288 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2pdkn" event={"ID":"2db677cb-ce90-46dd-b292-3d8c63da4fbb","Type":"ContainerStarted","Data":"74845fc9172e2961131a72386fb3a692796a0c7135ae0e83751df003893d6f28"} Jul 06 19:17:58.996044 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:58.995945 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2pdkn" event={"ID":"2db677cb-ce90-46dd-b292-3d8c63da4fbb","Type":"ContainerStarted","Data":"ad8f127f158bfaffded0d871bcbd92ed4743571f29d72523a710ad01114fc5bd"} Jul 06 19:17:58.996044 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:58.996000 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2pdkn" event={"ID":"2db677cb-ce90-46dd-b292-3d8c63da4fbb","Type":"ContainerStarted","Data":"427db9fb64198840551eabc37306bd1cae377ffef5b7d634c47841929b10d353"} Jul 06 19:17:59.014007 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:59.013952 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2pdkn" podStartSLOduration=374.030284462 podStartE2EDuration="6m15.013936072s" totalImagesPullingTime="983.65161ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:17:59.013237155 +0000 UTC m=+375.909126689" watchObservedRunningTime="2026-07-06 19:17:59.013936072 +0000 UTC m=+375.909825646" Jul 06 19:17:59.110872 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:59.110838 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:17:59.177600 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:17:59.177566 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:01.741487 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:18:01.741455 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:02.190559 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:18:02.190525 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:11.184582 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:18:11.184544 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:11.245136 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:18:11.245055 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:11.323837 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:18:11.323799 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:11.424679 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:18:11.424652 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:11.918609 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:18:11.918579 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:11.982209 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:18:11.982167 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:19.602645 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:18:19.602600 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:19.672004 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:18:19.671969 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:21.623843 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:18:21.623807 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:21.724872 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:18:21.724842 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:21.824029 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:18:21.823995 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:22.208069 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:18:22.208025 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:22.265363 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:18:22.265317 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:19:16.990056 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:19:16.990019 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:19:17.063592 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:19:17.063557 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:19:19.342062 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:19:19.342022 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:19:19.467904 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:19:19.467864 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:19:20.071582 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:19:20.071552 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:19:20.150273 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:19:20.150239 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:19:28.972851 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:19:28.972814 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:19:29.037918 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:19:29.037886 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:19:29.886916 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:19:29.886875 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:19:29.988874 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:19:29.988837 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:19:30.059815 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:19:30.059784 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:19:37.496293 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:19:37.496260 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:19:37.601169 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:19:37.601135 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:20:24.914837 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:20:24.914788 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:20:25.015783 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:20:25.015745 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:20:30.413798 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:20:30.413760 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:20:30.513781 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:20:30.513744 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:02.961783 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:02.961685 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:04.420182 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:04.420144 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:04.891824 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:04.891751 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wfmrs"] Jul 06 19:21:04.894830 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:04.894813 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wfmrs" Jul 06 19:21:04.897355 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:04.897334 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:21:04.897471 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:04.897374 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-sgcbd\"" Jul 06 19:21:04.897642 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:04.897626 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 06 19:21:04.900772 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:04.900753 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b03d30a2-ef89-490b-85c3-fc8122af140a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wfmrs\" (UID: \"b03d30a2-ef89-490b-85c3-fc8122af140a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wfmrs" Jul 06 19:21:04.900864 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:04.900789 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8wpx\" (UniqueName: \"kubernetes.io/projected/b03d30a2-ef89-490b-85c3-fc8122af140a-kube-api-access-s8wpx\") pod \"cert-manager-operator-controller-manager-55c444bf86-wfmrs\" (UID: \"b03d30a2-ef89-490b-85c3-fc8122af140a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wfmrs" Jul 06 19:21:04.917643 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:04.917614 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wfmrs"] Jul 06 19:21:05.001248 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:05.001210 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b03d30a2-ef89-490b-85c3-fc8122af140a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wfmrs\" (UID: \"b03d30a2-ef89-490b-85c3-fc8122af140a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wfmrs" Jul 06 19:21:05.001248 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:05.001259 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8wpx\" (UniqueName: \"kubernetes.io/projected/b03d30a2-ef89-490b-85c3-fc8122af140a-kube-api-access-s8wpx\") pod \"cert-manager-operator-controller-manager-55c444bf86-wfmrs\" (UID: \"b03d30a2-ef89-490b-85c3-fc8122af140a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wfmrs" Jul 06 19:21:05.001612 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:05.001591 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b03d30a2-ef89-490b-85c3-fc8122af140a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wfmrs\" (UID: \"b03d30a2-ef89-490b-85c3-fc8122af140a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wfmrs" Jul 06 19:21:05.017151 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:05.017117 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8wpx\" (UniqueName: \"kubernetes.io/projected/b03d30a2-ef89-490b-85c3-fc8122af140a-kube-api-access-s8wpx\") pod \"cert-manager-operator-controller-manager-55c444bf86-wfmrs\" (UID: \"b03d30a2-ef89-490b-85c3-fc8122af140a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wfmrs" Jul 06 19:21:05.204463 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:05.204424 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wfmrs" Jul 06 19:21:05.387025 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:05.386971 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wfmrs"] Jul 06 19:21:05.389411 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:21:05.389380 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03d30a2_ef89_490b_85c3_fc8122af140a.slice/crio-6c5849abf27912a8d4768b585804a731c45f317a7e1bc83ba4aaa6bd28e3fc0b WatchSource:0}: Error finding container 6c5849abf27912a8d4768b585804a731c45f317a7e1bc83ba4aaa6bd28e3fc0b: Status 404 returned error can't find the container with id 6c5849abf27912a8d4768b585804a731c45f317a7e1bc83ba4aaa6bd28e3fc0b Jul 06 19:21:05.626122 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:05.626037 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wfmrs" event={"ID":"b03d30a2-ef89-490b-85c3-fc8122af140a","Type":"ContainerStarted","Data":"6c5849abf27912a8d4768b585804a731c45f317a7e1bc83ba4aaa6bd28e3fc0b"} Jul 06 19:21:06.554603 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:06.554435 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:06.663799 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:06.663762 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:08.612165 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:08.612133 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:08.638966 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:08.638933 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wfmrs" event={"ID":"b03d30a2-ef89-490b-85c3-fc8122af140a","Type":"ContainerStarted","Data":"0f784ab803e8a456ddaa1b3207c955ee096bc50c3213f54123f58f8ea45342b7"} Jul 06 19:21:08.666349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:08.666304 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wfmrs" podStartSLOduration=2.210175031 podStartE2EDuration="4.666290205s" totalImagesPullingTime="2.456115174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:08.665695949 +0000 UTC m=+565.561585480" watchObservedRunningTime="2026-07-06 19:21:08.666290205 +0000 UTC m=+565.562179736" Jul 06 19:21:08.708992 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:08.708955 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:09.052310 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:09.052276 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:09.210939 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:09.210905 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:09.634244 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:09.634214 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489"] Jul 06 19:21:09.637430 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:09.637414 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489" Jul 06 19:21:09.639861 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:09.639840 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 19:21:09.640076 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:09.640063 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4wwgt\"" Jul 06 19:21:09.640828 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:09.640808 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 19:21:09.646231 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:09.646214 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489"] Jul 06 19:21:09.738166 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:09.738134 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489\" (UID: \"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489" Jul 06 19:21:09.738337 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:09.738174 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsblr\" (UniqueName: \"kubernetes.io/projected/e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7-kube-api-access-vsblr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489\" (UID: \"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489" Jul 06 19:21:09.738337 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:09.738209 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489\" (UID: \"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489" Jul 06 19:21:09.838985 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:09.838955 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489\" (UID: \"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489" Jul 06 19:21:09.838985 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:09.838992 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsblr\" (UniqueName: \"kubernetes.io/projected/e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7-kube-api-access-vsblr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489\" (UID: \"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489" Jul 06 19:21:09.839214 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:09.839018 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489\" (UID: \"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489" Jul 06 19:21:09.839367 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:09.839338 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489\" (UID: \"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489" Jul 06 19:21:09.839437 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:09.839421 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489\" (UID: \"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489" Jul 06 19:21:09.851349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:09.851320 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsblr\" (UniqueName: \"kubernetes.io/projected/e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7-kube-api-access-vsblr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489\" (UID: \"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489" Jul 06 19:21:09.947416 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:09.947382 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489" Jul 06 19:21:10.102357 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:10.102328 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489"] Jul 06 19:21:10.104386 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:21:10.104359 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e42e99_e1bb_4e4d_b3a0_701bc8ddd6a7.slice/crio-0d53e6af7742a20844523b771d5f63868cc3551b9f537e9e62dd52746b4b847a WatchSource:0}: Error finding container 0d53e6af7742a20844523b771d5f63868cc3551b9f537e9e62dd52746b4b847a: Status 404 returned error can't find the container with id 0d53e6af7742a20844523b771d5f63868cc3551b9f537e9e62dd52746b4b847a Jul 06 19:21:10.649127 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:10.649087 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489" event={"ID":"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7","Type":"ContainerStarted","Data":"0d53e6af7742a20844523b771d5f63868cc3551b9f537e9e62dd52746b4b847a"} Jul 06 19:21:10.811345 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:10.811310 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:10.987770 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:10.987732 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:13.533010 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:13.532977 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:13.589053 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:13.589021 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:18.820245 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:18.820189 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:20.071190 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:20.071156 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2fmfv"] Jul 06 19:21:20.075399 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:20.075372 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-2fmfv" Jul 06 19:21:20.078280 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:20.078258 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 19:21:20.078428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:20.078260 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-zm5nn\"" Jul 06 19:21:20.079055 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:20.079037 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 19:21:20.087291 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:20.087270 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2fmfv"] Jul 06 19:21:20.114762 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:20.114712 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5aa8d6f-2b97-4327-ad79-3840fcbdcee1-bound-sa-token\") pod \"cert-manager-55b4d954d5-2fmfv\" (UID: \"c5aa8d6f-2b97-4327-ad79-3840fcbdcee1\") " pod="cert-manager/cert-manager-55b4d954d5-2fmfv" Jul 06 19:21:20.114942 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:20.114847 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlccc\" (UniqueName: \"kubernetes.io/projected/c5aa8d6f-2b97-4327-ad79-3840fcbdcee1-kube-api-access-xlccc\") pod \"cert-manager-55b4d954d5-2fmfv\" (UID: \"c5aa8d6f-2b97-4327-ad79-3840fcbdcee1\") " pod="cert-manager/cert-manager-55b4d954d5-2fmfv" Jul 06 19:21:20.215447 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:20.215404 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlccc\" (UniqueName: \"kubernetes.io/projected/c5aa8d6f-2b97-4327-ad79-3840fcbdcee1-kube-api-access-xlccc\") pod \"cert-manager-55b4d954d5-2fmfv\" (UID: \"c5aa8d6f-2b97-4327-ad79-3840fcbdcee1\") " pod="cert-manager/cert-manager-55b4d954d5-2fmfv" Jul 06 19:21:20.215635 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:20.215476 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5aa8d6f-2b97-4327-ad79-3840fcbdcee1-bound-sa-token\") pod \"cert-manager-55b4d954d5-2fmfv\" (UID: \"c5aa8d6f-2b97-4327-ad79-3840fcbdcee1\") " pod="cert-manager/cert-manager-55b4d954d5-2fmfv" Jul 06 19:21:20.224631 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:20.224602 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5aa8d6f-2b97-4327-ad79-3840fcbdcee1-bound-sa-token\") pod \"cert-manager-55b4d954d5-2fmfv\" (UID: \"c5aa8d6f-2b97-4327-ad79-3840fcbdcee1\") " pod="cert-manager/cert-manager-55b4d954d5-2fmfv" Jul 06 19:21:20.224797 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:20.224697 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlccc\" (UniqueName: \"kubernetes.io/projected/c5aa8d6f-2b97-4327-ad79-3840fcbdcee1-kube-api-access-xlccc\") pod \"cert-manager-55b4d954d5-2fmfv\" (UID: \"c5aa8d6f-2b97-4327-ad79-3840fcbdcee1\") " pod="cert-manager/cert-manager-55b4d954d5-2fmfv" Jul 06 19:21:20.385844 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:20.385748 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-2fmfv" Jul 06 19:21:20.622003 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:20.621962 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:20.693906 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:20.693871 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:21.592914 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:21.592891 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2fmfv"] Jul 06 19:21:21.596043 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:21:21.596018 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5aa8d6f_2b97_4327_ad79_3840fcbdcee1.slice/crio-6449bf1710a849f964fdcd9df9ad1d127883d32eb06413ab6cfa69a0a995e1af WatchSource:0}: Error finding container 6449bf1710a849f964fdcd9df9ad1d127883d32eb06413ab6cfa69a0a995e1af: Status 404 returned error can't find the container with id 6449bf1710a849f964fdcd9df9ad1d127883d32eb06413ab6cfa69a0a995e1af Jul 06 19:21:21.690590 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:21.690556 2566 generic.go:356] "Generic (PLEG): container finished" podID="e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7" containerID="7f7955bf3efdf6974cea04bccd87a8d54b9b2d7c2be759425d39c2e0c1ba097c" exitCode=0 Jul 06 19:21:21.690749 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:21.690645 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489" event={"ID":"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7","Type":"ContainerDied","Data":"7f7955bf3efdf6974cea04bccd87a8d54b9b2d7c2be759425d39c2e0c1ba097c"} Jul 06 19:21:21.694105 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:21.692473 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-2fmfv" event={"ID":"c5aa8d6f-2b97-4327-ad79-3840fcbdcee1","Type":"ContainerStarted","Data":"6449bf1710a849f964fdcd9df9ad1d127883d32eb06413ab6cfa69a0a995e1af"} Jul 06 19:21:23.146706 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:23.146669 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:23.703899 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:23.703861 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489" event={"ID":"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7","Type":"ContainerStarted","Data":"a776e13fa59709d3a80d043d2b92d7e31385a1065a70e7aca7aa3c4604472491"} Jul 06 19:21:24.710640 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:24.710606 2566 generic.go:356] "Generic (PLEG): container finished" podID="e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7" containerID="a776e13fa59709d3a80d043d2b92d7e31385a1065a70e7aca7aa3c4604472491" exitCode=0 Jul 06 19:21:24.711073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:24.710669 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489" event={"ID":"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7","Type":"ContainerDied","Data":"a776e13fa59709d3a80d043d2b92d7e31385a1065a70e7aca7aa3c4604472491"} Jul 06 19:21:25.716174 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:25.716133 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-2fmfv" event={"ID":"c5aa8d6f-2b97-4327-ad79-3840fcbdcee1","Type":"ContainerStarted","Data":"e236b8f15bad750174813ab75852c733239546c3865f4ca30353e261cd8237a0"} Jul 06 19:21:25.735741 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:25.735676 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-2fmfv" podStartSLOduration=2.219145797 podStartE2EDuration="5.735660621s" totalImagesPullingTime="3.516514824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:25.73506324 +0000 UTC m=+582.630952774" watchObservedRunningTime="2026-07-06 19:21:25.735660621 +0000 UTC m=+582.631550154" Jul 06 19:21:31.701280 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:31.701241 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:33.749772 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:33.749740 2566 generic.go:356] "Generic (PLEG): container finished" podID="e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7" containerID="5ffc2ad6199bf007211e71fbb9382cd490455d1cd80f70eb6d384022181b889d" exitCode=0 Jul 06 19:21:33.750280 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:33.749821 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489" event={"ID":"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7","Type":"ContainerDied","Data":"5ffc2ad6199bf007211e71fbb9382cd490455d1cd80f70eb6d384022181b889d"} Jul 06 19:21:34.184653 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:34.184625 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:34.874149 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:34.874122 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489" Jul 06 19:21:34.933562 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:34.933534 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7-util\") pod \"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7\" (UID: \"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7\") " Jul 06 19:21:34.933759 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:34.933584 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsblr\" (UniqueName: \"kubernetes.io/projected/e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7-kube-api-access-vsblr\") pod \"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7\" (UID: \"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7\") " Jul 06 19:21:34.933759 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:34.933635 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7-bundle\") pod \"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7\" (UID: \"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7\") " Jul 06 19:21:34.934104 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:34.934076 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7-bundle" (OuterVolumeSpecName: "bundle") pod "e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7" (UID: "e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:21:34.935775 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:34.935750 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7-kube-api-access-vsblr" (OuterVolumeSpecName: "kube-api-access-vsblr") pod "e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7" (UID: "e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7"). InnerVolumeSpecName "kube-api-access-vsblr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:21:34.937653 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:34.937628 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7-util" (OuterVolumeSpecName: "util") pod "e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7" (UID: "e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:21:35.034758 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:35.034632 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7-util\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:21:35.034758 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:35.034676 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsblr\" (UniqueName: \"kubernetes.io/projected/e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7-kube-api-access-vsblr\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:21:35.034758 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:35.034686 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7-bundle\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:21:35.760051 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:35.760025 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489" Jul 06 19:21:35.762374 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:35.762342 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frq489" event={"ID":"e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7","Type":"ContainerDied","Data":"0d53e6af7742a20844523b771d5f63868cc3551b9f537e9e62dd52746b4b847a"} Jul 06 19:21:35.762484 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:35.762388 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d53e6af7742a20844523b771d5f63868cc3551b9f537e9e62dd52746b4b847a" Jul 06 19:21:38.573617 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:38.573581 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:38.674653 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:38.674615 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:39.996173 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:39.996137 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgs8p"] Jul 06 19:21:39.996625 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:39.996414 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7" containerName="extract" Jul 06 19:21:39.996625 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:39.996426 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7" containerName="extract" Jul 06 19:21:39.996625 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:39.996436 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7" containerName="util" Jul 06 19:21:39.996625 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:39.996441 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7" containerName="util" Jul 06 19:21:39.996625 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:39.996449 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7" containerName="pull" Jul 06 19:21:39.996625 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:39.996457 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7" containerName="pull" Jul 06 19:21:39.996625 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:39.996503 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4e42e99-e1bb-4e4d-b3a0-701bc8ddd6a7" containerName="extract" Jul 06 19:21:40.041000 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:40.040960 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgs8p"] Jul 06 19:21:40.041159 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:40.041077 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgs8p" Jul 06 19:21:40.043899 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:40.043879 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:21:40.044539 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:40.044511 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-fph87\"" Jul 06 19:21:40.044539 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:40.044511 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 19:21:40.176002 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:40.175965 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eab4be20-1503-4405-8541-0ecb2172a74e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tgs8p\" (UID: \"eab4be20-1503-4405-8541-0ecb2172a74e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgs8p" Jul 06 19:21:40.176180 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:40.176023 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ggtz\" (UniqueName: \"kubernetes.io/projected/eab4be20-1503-4405-8541-0ecb2172a74e-kube-api-access-7ggtz\") pod \"openshift-lws-operator-fd8ccff4c-tgs8p\" (UID: \"eab4be20-1503-4405-8541-0ecb2172a74e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgs8p" Jul 06 19:21:40.277324 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:40.277236 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eab4be20-1503-4405-8541-0ecb2172a74e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tgs8p\" (UID: \"eab4be20-1503-4405-8541-0ecb2172a74e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgs8p" Jul 06 19:21:40.277324 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:40.277289 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ggtz\" (UniqueName: \"kubernetes.io/projected/eab4be20-1503-4405-8541-0ecb2172a74e-kube-api-access-7ggtz\") pod \"openshift-lws-operator-fd8ccff4c-tgs8p\" (UID: \"eab4be20-1503-4405-8541-0ecb2172a74e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgs8p" Jul 06 19:21:40.277676 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:40.277652 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eab4be20-1503-4405-8541-0ecb2172a74e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tgs8p\" (UID: \"eab4be20-1503-4405-8541-0ecb2172a74e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgs8p" Jul 06 19:21:40.286111 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:40.286085 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ggtz\" (UniqueName: \"kubernetes.io/projected/eab4be20-1503-4405-8541-0ecb2172a74e-kube-api-access-7ggtz\") pod \"openshift-lws-operator-fd8ccff4c-tgs8p\" (UID: \"eab4be20-1503-4405-8541-0ecb2172a74e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgs8p" Jul 06 19:21:40.350606 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:40.350558 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgs8p" Jul 06 19:21:40.499945 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:40.499763 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgs8p"] Jul 06 19:21:40.502452 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:21:40.502425 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeab4be20_1503_4405_8541_0ecb2172a74e.slice/crio-b93ce7a290a96ec0caf53d51872eb2898471430a4ec70334c4f42f838f578456 WatchSource:0}: Error finding container b93ce7a290a96ec0caf53d51872eb2898471430a4ec70334c4f42f838f578456: Status 404 returned error can't find the container with id b93ce7a290a96ec0caf53d51872eb2898471430a4ec70334c4f42f838f578456 Jul 06 19:21:40.779383 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:40.779354 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgs8p" event={"ID":"eab4be20-1503-4405-8541-0ecb2172a74e","Type":"ContainerStarted","Data":"b93ce7a290a96ec0caf53d51872eb2898471430a4ec70334c4f42f838f578456"} Jul 06 19:21:43.557695 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:43.557672 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:21:43.558134 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:43.557705 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:21:53.275080 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:53.275040 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:53.375898 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:53.375856 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:55.139194 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:55.139159 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:21:55.209910 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:21:55.209864 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:07.885295 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:07.885258 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgs8p" event={"ID":"eab4be20-1503-4405-8541-0ecb2172a74e","Type":"ContainerStarted","Data":"f6cf92949416c2153268395eb5a84311876dde849da3243bd3619e35e65511fd"} Jul 06 19:22:07.903396 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:07.903346 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgs8p" podStartSLOduration=2.199121472 podStartE2EDuration="28.903330136s" totalImagesPullingTime="26.704208664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:07.901231665 +0000 UTC m=+624.797121198" watchObservedRunningTime="2026-07-06 19:22:07.903330136 +0000 UTC m=+624.799219668" Jul 06 19:22:08.578515 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:08.578480 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:08.678140 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:08.678087 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:10.706314 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:10.706269 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:15.096334 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:15.096291 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:15.195081 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:15.195051 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:25.170637 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:25.170600 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:25.291312 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:25.291276 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:25.806552 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:25.806514 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:26.044378 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.044319 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:26.154348 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.154256 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:26.238291 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.238260 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg"] Jul 06 19:22:26.240420 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.240407 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" Jul 06 19:22:26.245121 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.245101 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 19:22:26.245252 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.245141 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 19:22:26.245319 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.245290 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 19:22:26.246275 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.246252 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 19:22:26.246430 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.246408 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jdd5t\"" Jul 06 19:22:26.282514 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.282483 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg"] Jul 06 19:22:26.311861 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.311826 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ca071df-c56c-4c75-bd57-f30e0b31976f-manifests\") pod \"opendatahub-operator-controller-manager-56f978bf57-w7bsg\" (UID: \"4ca071df-c56c-4c75-bd57-f30e0b31976f\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" Jul 06 19:22:26.311861 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.311866 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ca071df-c56c-4c75-bd57-f30e0b31976f-tmp\") pod \"opendatahub-operator-controller-manager-56f978bf57-w7bsg\" (UID: \"4ca071df-c56c-4c75-bd57-f30e0b31976f\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" Jul 06 19:22:26.312078 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.311930 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ca071df-c56c-4c75-bd57-f30e0b31976f-webhook-cert\") pod \"opendatahub-operator-controller-manager-56f978bf57-w7bsg\" (UID: \"4ca071df-c56c-4c75-bd57-f30e0b31976f\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" Jul 06 19:22:26.312078 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.311990 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ca071df-c56c-4c75-bd57-f30e0b31976f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56f978bf57-w7bsg\" (UID: \"4ca071df-c56c-4c75-bd57-f30e0b31976f\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" Jul 06 19:22:26.312078 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.312024 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbw7z\" (UniqueName: \"kubernetes.io/projected/4ca071df-c56c-4c75-bd57-f30e0b31976f-kube-api-access-vbw7z\") pod \"opendatahub-operator-controller-manager-56f978bf57-w7bsg\" (UID: \"4ca071df-c56c-4c75-bd57-f30e0b31976f\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" Jul 06 19:22:26.405929 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.405838 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:26.412836 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.412814 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ca071df-c56c-4c75-bd57-f30e0b31976f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56f978bf57-w7bsg\" (UID: \"4ca071df-c56c-4c75-bd57-f30e0b31976f\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" Jul 06 19:22:26.412956 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.412851 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbw7z\" (UniqueName: \"kubernetes.io/projected/4ca071df-c56c-4c75-bd57-f30e0b31976f-kube-api-access-vbw7z\") pod \"opendatahub-operator-controller-manager-56f978bf57-w7bsg\" (UID: \"4ca071df-c56c-4c75-bd57-f30e0b31976f\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" Jul 06 19:22:26.412956 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.412893 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ca071df-c56c-4c75-bd57-f30e0b31976f-manifests\") pod \"opendatahub-operator-controller-manager-56f978bf57-w7bsg\" (UID: \"4ca071df-c56c-4c75-bd57-f30e0b31976f\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" Jul 06 19:22:26.412956 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.412920 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ca071df-c56c-4c75-bd57-f30e0b31976f-tmp\") pod \"opendatahub-operator-controller-manager-56f978bf57-w7bsg\" (UID: \"4ca071df-c56c-4c75-bd57-f30e0b31976f\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" Jul 06 19:22:26.413066 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.412960 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ca071df-c56c-4c75-bd57-f30e0b31976f-webhook-cert\") pod \"opendatahub-operator-controller-manager-56f978bf57-w7bsg\" (UID: \"4ca071df-c56c-4c75-bd57-f30e0b31976f\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" Jul 06 19:22:26.413396 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.413375 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ca071df-c56c-4c75-bd57-f30e0b31976f-manifests\") pod \"opendatahub-operator-controller-manager-56f978bf57-w7bsg\" (UID: \"4ca071df-c56c-4c75-bd57-f30e0b31976f\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" Jul 06 19:22:26.415374 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.415351 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ca071df-c56c-4c75-bd57-f30e0b31976f-tmp\") pod \"opendatahub-operator-controller-manager-56f978bf57-w7bsg\" (UID: \"4ca071df-c56c-4c75-bd57-f30e0b31976f\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" Jul 06 19:22:26.415507 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.415486 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ca071df-c56c-4c75-bd57-f30e0b31976f-webhook-cert\") pod \"opendatahub-operator-controller-manager-56f978bf57-w7bsg\" (UID: \"4ca071df-c56c-4c75-bd57-f30e0b31976f\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" Jul 06 19:22:26.415553 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.415506 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ca071df-c56c-4c75-bd57-f30e0b31976f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56f978bf57-w7bsg\" (UID: \"4ca071df-c56c-4c75-bd57-f30e0b31976f\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" Jul 06 19:22:26.424573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.424553 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbw7z\" (UniqueName: \"kubernetes.io/projected/4ca071df-c56c-4c75-bd57-f30e0b31976f-kube-api-access-vbw7z\") pod \"opendatahub-operator-controller-manager-56f978bf57-w7bsg\" (UID: \"4ca071df-c56c-4c75-bd57-f30e0b31976f\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" Jul 06 19:22:26.506094 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.506058 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:26.550128 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.550094 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" Jul 06 19:22:26.609771 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.609745 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:26.694305 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.694276 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg"] Jul 06 19:22:26.697576 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:22:26.697549 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca071df_c56c_4c75_bd57_f30e0b31976f.slice/crio-4b9e5b9f128f690feb9f8f87f3b882b34b42e64c7d826483a6546a53c7f134bc WatchSource:0}: Error finding container 4b9e5b9f128f690feb9f8f87f3b882b34b42e64c7d826483a6546a53c7f134bc: Status 404 returned error can't find the container with id 4b9e5b9f128f690feb9f8f87f3b882b34b42e64c7d826483a6546a53c7f134bc Jul 06 19:22:26.699379 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.699364 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:22:26.701017 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.700996 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:26.772656 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.772622 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:26.877162 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.877131 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:26.973227 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:26.973080 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" event={"ID":"4ca071df-c56c-4c75-bd57-f30e0b31976f","Type":"ContainerStarted","Data":"4b9e5b9f128f690feb9f8f87f3b882b34b42e64c7d826483a6546a53c7f134bc"} Jul 06 19:22:31.996132 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:31.996091 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" event={"ID":"4ca071df-c56c-4c75-bd57-f30e0b31976f","Type":"ContainerStarted","Data":"307eb2517757d2b32c169afd1068f42132a6cbd180da39b46f64d9bfd30a5ada"} Jul 06 19:22:33.001416 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:33.001380 2566 generic.go:356] "Generic (PLEG): container finished" podID="4ca071df-c56c-4c75-bd57-f30e0b31976f" containerID="307eb2517757d2b32c169afd1068f42132a6cbd180da39b46f64d9bfd30a5ada" exitCode=0 Jul 06 19:22:33.001825 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:33.001426 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" event={"ID":"4ca071df-c56c-4c75-bd57-f30e0b31976f","Type":"ContainerDied","Data":"307eb2517757d2b32c169afd1068f42132a6cbd180da39b46f64d9bfd30a5ada"} Jul 06 19:22:34.008058 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:34.008025 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" event={"ID":"4ca071df-c56c-4c75-bd57-f30e0b31976f","Type":"ContainerStarted","Data":"31f5218311c7f53988dc0abe41b0d370290fc56d27b64c1c965116eb58e6e149"} Jul 06 19:22:34.008492 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:34.008158 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" Jul 06 19:22:34.030426 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:34.030375 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" podStartSLOduration=1.8869010959999999 podStartE2EDuration="8.030360101s" totalImagesPullingTime="5.075468892s" totalInitContainerRuntime="1.067990113s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:34.029078969 +0000 UTC m=+650.924968502" watchObservedRunningTime="2026-07-06 19:22:34.030360101 +0000 UTC m=+650.926249633" Jul 06 19:22:45.014352 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:45.014319 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-w7bsg" Jul 06 19:22:47.073577 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.073527 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:47.235478 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.235434 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:47.546932 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.546894 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:47.574843 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.574798 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6fffdf4997-n76hp"] Jul 06 19:22:47.579547 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.579525 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fffdf4997-n76hp" Jul 06 19:22:47.582484 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.582465 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 19:22:47.583125 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.583109 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 19:22:47.583193 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.583154 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 19:22:47.589440 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.589425 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 19:22:47.595574 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.595553 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fffdf4997-n76hp"] Jul 06 19:22:47.667964 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.667934 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b354c43c-54c3-4679-9bc3-dea447406712-tmp\") pod \"kube-auth-proxy-6fffdf4997-n76hp\" (UID: \"b354c43c-54c3-4679-9bc3-dea447406712\") " pod="openshift-ingress/kube-auth-proxy-6fffdf4997-n76hp" Jul 06 19:22:47.668140 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.667987 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmfq2\" (UniqueName: \"kubernetes.io/projected/b354c43c-54c3-4679-9bc3-dea447406712-kube-api-access-wmfq2\") pod \"kube-auth-proxy-6fffdf4997-n76hp\" (UID: \"b354c43c-54c3-4679-9bc3-dea447406712\") " pod="openshift-ingress/kube-auth-proxy-6fffdf4997-n76hp" Jul 06 19:22:47.668140 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.668075 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b354c43c-54c3-4679-9bc3-dea447406712-tls-certs\") pod \"kube-auth-proxy-6fffdf4997-n76hp\" (UID: \"b354c43c-54c3-4679-9bc3-dea447406712\") " pod="openshift-ingress/kube-auth-proxy-6fffdf4997-n76hp" Jul 06 19:22:47.708117 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.708083 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:47.768857 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.768828 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b354c43c-54c3-4679-9bc3-dea447406712-tmp\") pod \"kube-auth-proxy-6fffdf4997-n76hp\" (UID: \"b354c43c-54c3-4679-9bc3-dea447406712\") " pod="openshift-ingress/kube-auth-proxy-6fffdf4997-n76hp" Jul 06 19:22:47.769035 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.768873 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmfq2\" (UniqueName: \"kubernetes.io/projected/b354c43c-54c3-4679-9bc3-dea447406712-kube-api-access-wmfq2\") pod \"kube-auth-proxy-6fffdf4997-n76hp\" (UID: \"b354c43c-54c3-4679-9bc3-dea447406712\") " pod="openshift-ingress/kube-auth-proxy-6fffdf4997-n76hp" Jul 06 19:22:47.769104 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.769027 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b354c43c-54c3-4679-9bc3-dea447406712-tls-certs\") pod \"kube-auth-proxy-6fffdf4997-n76hp\" (UID: \"b354c43c-54c3-4679-9bc3-dea447406712\") " pod="openshift-ingress/kube-auth-proxy-6fffdf4997-n76hp" Jul 06 19:22:47.771279 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.771257 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b354c43c-54c3-4679-9bc3-dea447406712-tmp\") pod \"kube-auth-proxy-6fffdf4997-n76hp\" (UID: \"b354c43c-54c3-4679-9bc3-dea447406712\") " pod="openshift-ingress/kube-auth-proxy-6fffdf4997-n76hp" Jul 06 19:22:47.771397 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.771382 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b354c43c-54c3-4679-9bc3-dea447406712-tls-certs\") pod \"kube-auth-proxy-6fffdf4997-n76hp\" (UID: \"b354c43c-54c3-4679-9bc3-dea447406712\") " pod="openshift-ingress/kube-auth-proxy-6fffdf4997-n76hp" Jul 06 19:22:47.784430 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.784410 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmfq2\" (UniqueName: \"kubernetes.io/projected/b354c43c-54c3-4679-9bc3-dea447406712-kube-api-access-wmfq2\") pod \"kube-auth-proxy-6fffdf4997-n76hp\" (UID: \"b354c43c-54c3-4679-9bc3-dea447406712\") " pod="openshift-ingress/kube-auth-proxy-6fffdf4997-n76hp" Jul 06 19:22:47.890130 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:47.890035 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fffdf4997-n76hp" Jul 06 19:22:48.051879 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:48.051777 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fffdf4997-n76hp"] Jul 06 19:22:48.054418 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:22:48.054381 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb354c43c_54c3_4679_9bc3_dea447406712.slice/crio-9c977efb420f933b2aff2b55fde95e48d6eec2fbc43fd6986a86245af2fa63bc WatchSource:0}: Error finding container 9c977efb420f933b2aff2b55fde95e48d6eec2fbc43fd6986a86245af2fa63bc: Status 404 returned error can't find the container with id 9c977efb420f933b2aff2b55fde95e48d6eec2fbc43fd6986a86245af2fa63bc Jul 06 19:22:48.068271 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:48.068239 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fffdf4997-n76hp" event={"ID":"b354c43c-54c3-4679-9bc3-dea447406712","Type":"ContainerStarted","Data":"9c977efb420f933b2aff2b55fde95e48d6eec2fbc43fd6986a86245af2fa63bc"} Jul 06 19:22:48.142857 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:48.142771 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lf8hn"] Jul 06 19:22:48.147261 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:48.147239 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lf8hn" Jul 06 19:22:48.151216 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:48.151197 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 06 19:22:48.151481 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:48.151468 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qj4gf\"" Jul 06 19:22:48.165399 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:48.165375 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lf8hn"] Jul 06 19:22:48.172884 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:48.172856 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33a0983c-34cc-4244-99d0-7e3e69f94bbc-cert\") pod \"odh-model-controller-7998c67dc4-lf8hn\" (UID: \"33a0983c-34cc-4244-99d0-7e3e69f94bbc\") " pod="opendatahub/odh-model-controller-7998c67dc4-lf8hn" Jul 06 19:22:48.173002 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:48.172943 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zrp5\" (UniqueName: \"kubernetes.io/projected/33a0983c-34cc-4244-99d0-7e3e69f94bbc-kube-api-access-7zrp5\") pod \"odh-model-controller-7998c67dc4-lf8hn\" (UID: \"33a0983c-34cc-4244-99d0-7e3e69f94bbc\") " pod="opendatahub/odh-model-controller-7998c67dc4-lf8hn" Jul 06 19:22:48.273975 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:48.273947 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zrp5\" (UniqueName: \"kubernetes.io/projected/33a0983c-34cc-4244-99d0-7e3e69f94bbc-kube-api-access-7zrp5\") pod \"odh-model-controller-7998c67dc4-lf8hn\" (UID: \"33a0983c-34cc-4244-99d0-7e3e69f94bbc\") " pod="opendatahub/odh-model-controller-7998c67dc4-lf8hn" Jul 06 19:22:48.274156 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:48.274029 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33a0983c-34cc-4244-99d0-7e3e69f94bbc-cert\") pod \"odh-model-controller-7998c67dc4-lf8hn\" (UID: \"33a0983c-34cc-4244-99d0-7e3e69f94bbc\") " pod="opendatahub/odh-model-controller-7998c67dc4-lf8hn" Jul 06 19:22:48.274222 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:22:48.274180 2566 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 06 19:22:48.274278 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:22:48.274257 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33a0983c-34cc-4244-99d0-7e3e69f94bbc-cert podName:33a0983c-34cc-4244-99d0-7e3e69f94bbc nodeName:}" failed. No retries permitted until 2026-07-06 19:22:48.774235344 +0000 UTC m=+665.670124872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/33a0983c-34cc-4244-99d0-7e3e69f94bbc-cert") pod "odh-model-controller-7998c67dc4-lf8hn" (UID: "33a0983c-34cc-4244-99d0-7e3e69f94bbc") : secret "odh-model-controller-webhook-cert" not found Jul 06 19:22:48.279787 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:48.279756 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:48.286206 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:48.286183 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zrp5\" (UniqueName: \"kubernetes.io/projected/33a0983c-34cc-4244-99d0-7e3e69f94bbc-kube-api-access-7zrp5\") pod \"odh-model-controller-7998c67dc4-lf8hn\" (UID: \"33a0983c-34cc-4244-99d0-7e3e69f94bbc\") " pod="opendatahub/odh-model-controller-7998c67dc4-lf8hn" Jul 06 19:22:48.386216 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:48.386180 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:48.506042 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:48.506012 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:48.606537 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:48.606501 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:48.781441 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:48.781362 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33a0983c-34cc-4244-99d0-7e3e69f94bbc-cert\") pod \"odh-model-controller-7998c67dc4-lf8hn\" (UID: \"33a0983c-34cc-4244-99d0-7e3e69f94bbc\") " pod="opendatahub/odh-model-controller-7998c67dc4-lf8hn" Jul 06 19:22:48.786948 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:48.786917 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33a0983c-34cc-4244-99d0-7e3e69f94bbc-cert\") pod \"odh-model-controller-7998c67dc4-lf8hn\" (UID: \"33a0983c-34cc-4244-99d0-7e3e69f94bbc\") " pod="opendatahub/odh-model-controller-7998c67dc4-lf8hn" Jul 06 19:22:49.057806 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.057700 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lf8hn" Jul 06 19:22:49.187801 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.187769 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-d2scm"] Jul 06 19:22:49.191255 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.191239 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-d2scm" Jul 06 19:22:49.193580 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.193561 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 06 19:22:49.202938 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.202918 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-d2scm"] Jul 06 19:22:49.204232 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.204205 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lf8hn"] Jul 06 19:22:49.206467 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:22:49.206441 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a0983c_34cc_4244_99d0_7e3e69f94bbc.slice/crio-27ad1d5ad7fb2147344efefb12620195e6b5178349f8324ed208ea73c813493a WatchSource:0}: Error finding container 27ad1d5ad7fb2147344efefb12620195e6b5178349f8324ed208ea73c813493a: Status 404 returned error can't find the container with id 27ad1d5ad7fb2147344efefb12620195e6b5178349f8324ed208ea73c813493a Jul 06 19:22:49.227496 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.227454 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:49.286126 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.286083 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e5da542-3f91-4f05-8959-c4c05f9210a9-tls-certs\") pod \"model-serving-api-54d89bd5fc-d2scm\" (UID: \"0e5da542-3f91-4f05-8959-c4c05f9210a9\") " pod="opendatahub/model-serving-api-54d89bd5fc-d2scm" Jul 06 19:22:49.286314 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.286204 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7xz7\" (UniqueName: \"kubernetes.io/projected/0e5da542-3f91-4f05-8959-c4c05f9210a9-kube-api-access-f7xz7\") pod \"model-serving-api-54d89bd5fc-d2scm\" (UID: \"0e5da542-3f91-4f05-8959-c4c05f9210a9\") " pod="opendatahub/model-serving-api-54d89bd5fc-d2scm" Jul 06 19:22:49.318690 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.318610 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-grwnb"] Jul 06 19:22:49.322248 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.322229 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.324712 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.324666 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 06 19:22:49.324845 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.324763 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 06 19:22:49.324845 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.324786 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-m5m87\"" Jul 06 19:22:49.324845 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.324839 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 19:22:49.325073 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.325036 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 06 19:22:49.326307 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.326280 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:49.338326 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.338301 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-grwnb"] Jul 06 19:22:49.386821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.386786 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e79c825d-53e2-41e5-b758-eef94b52e1fa-cacerts\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.386821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.386818 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.387029 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.386850 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e5da542-3f91-4f05-8959-c4c05f9210a9-tls-certs\") pod \"model-serving-api-54d89bd5fc-d2scm\" (UID: \"0e5da542-3f91-4f05-8959-c4c05f9210a9\") " pod="opendatahub/model-serving-api-54d89bd5fc-d2scm" Jul 06 19:22:49.387029 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.386930 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qwl2\" (UniqueName: \"kubernetes.io/projected/e79c825d-53e2-41e5-b758-eef94b52e1fa-kube-api-access-5qwl2\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.387029 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:22:49.386944 2566 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 06 19:22:49.387029 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.386962 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e79c825d-53e2-41e5-b758-eef94b52e1fa-local-certs\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.387029 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.386988 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-token\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.387029 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:22:49.387002 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e5da542-3f91-4f05-8959-c4c05f9210a9-tls-certs podName:0e5da542-3f91-4f05-8959-c4c05f9210a9 nodeName:}" failed. No retries permitted until 2026-07-06 19:22:49.886985792 +0000 UTC m=+666.782875301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0e5da542-3f91-4f05-8959-c4c05f9210a9-tls-certs") pod "model-serving-api-54d89bd5fc-d2scm" (UID: "0e5da542-3f91-4f05-8959-c4c05f9210a9") : secret "model-serving-api-tls" not found Jul 06 19:22:49.387231 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.387057 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7xz7\" (UniqueName: \"kubernetes.io/projected/0e5da542-3f91-4f05-8959-c4c05f9210a9-kube-api-access-f7xz7\") pod \"model-serving-api-54d89bd5fc-d2scm\" (UID: \"0e5da542-3f91-4f05-8959-c4c05f9210a9\") " pod="opendatahub/model-serving-api-54d89bd5fc-d2scm" Jul 06 19:22:49.387231 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.387084 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.387231 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.387126 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.397490 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.397466 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7xz7\" (UniqueName: \"kubernetes.io/projected/0e5da542-3f91-4f05-8959-c4c05f9210a9-kube-api-access-f7xz7\") pod \"model-serving-api-54d89bd5fc-d2scm\" (UID: \"0e5da542-3f91-4f05-8959-c4c05f9210a9\") " pod="opendatahub/model-serving-api-54d89bd5fc-d2scm" Jul 06 19:22:49.426206 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.426159 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:49.487890 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.487858 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qwl2\" (UniqueName: \"kubernetes.io/projected/e79c825d-53e2-41e5-b758-eef94b52e1fa-kube-api-access-5qwl2\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.488070 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.487905 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e79c825d-53e2-41e5-b758-eef94b52e1fa-local-certs\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.488070 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.487936 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-token\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.488070 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.487968 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.488070 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.488014 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.488070 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.488043 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e79c825d-53e2-41e5-b758-eef94b52e1fa-cacerts\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.488070 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.488066 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.488983 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.488935 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.491119 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.491094 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e79c825d-53e2-41e5-b758-eef94b52e1fa-local-certs\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.491252 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.491093 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.491252 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.491128 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e79c825d-53e2-41e5-b758-eef94b52e1fa-cacerts\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.491362 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.491276 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.496061 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.496036 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qwl2\" (UniqueName: \"kubernetes.io/projected/e79c825d-53e2-41e5-b758-eef94b52e1fa-kube-api-access-5qwl2\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.496487 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.496450 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-token\") pod \"istiod-openshift-gateway-7574459656-grwnb\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.632990 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.632907 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:49.728077 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.728041 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:49.772869 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.772843 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-grwnb"] Jul 06 19:22:49.774889 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:22:49.774858 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79c825d_53e2_41e5_b758_eef94b52e1fa.slice/crio-468a41ba4954100788590821fe526e592f8551f613adb7c4b3f27c1255e5b8a0 WatchSource:0}: Error finding container 468a41ba4954100788590821fe526e592f8551f613adb7c4b3f27c1255e5b8a0: Status 404 returned error can't find the container with id 468a41ba4954100788590821fe526e592f8551f613adb7c4b3f27c1255e5b8a0 Jul 06 19:22:49.891780 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.891683 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e5da542-3f91-4f05-8959-c4c05f9210a9-tls-certs\") pod \"model-serving-api-54d89bd5fc-d2scm\" (UID: \"0e5da542-3f91-4f05-8959-c4c05f9210a9\") " pod="opendatahub/model-serving-api-54d89bd5fc-d2scm" Jul 06 19:22:49.894201 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:49.894171 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e5da542-3f91-4f05-8959-c4c05f9210a9-tls-certs\") pod \"model-serving-api-54d89bd5fc-d2scm\" (UID: \"0e5da542-3f91-4f05-8959-c4c05f9210a9\") " pod="opendatahub/model-serving-api-54d89bd5fc-d2scm" Jul 06 19:22:50.080509 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:50.080471 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" event={"ID":"e79c825d-53e2-41e5-b758-eef94b52e1fa","Type":"ContainerStarted","Data":"468a41ba4954100788590821fe526e592f8551f613adb7c4b3f27c1255e5b8a0"} Jul 06 19:22:50.081688 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:50.081666 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lf8hn" event={"ID":"33a0983c-34cc-4244-99d0-7e3e69f94bbc","Type":"ContainerStarted","Data":"27ad1d5ad7fb2147344efefb12620195e6b5178349f8324ed208ea73c813493a"} Jul 06 19:22:50.102077 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:50.102025 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-d2scm" Jul 06 19:22:50.272213 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:50.272189 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-d2scm"] Jul 06 19:22:50.275023 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:22:50.274987 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5da542_3f91_4f05_8959_c4c05f9210a9.slice/crio-eba5bddee02bf1559420b3d28587ba43cbf8f4e226501b94a4b10cbc06527f0e WatchSource:0}: Error finding container eba5bddee02bf1559420b3d28587ba43cbf8f4e226501b94a4b10cbc06527f0e: Status 404 returned error can't find the container with id eba5bddee02bf1559420b3d28587ba43cbf8f4e226501b94a4b10cbc06527f0e Jul 06 19:22:50.484502 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:50.484422 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:51.091350 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:51.091275 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-d2scm" event={"ID":"0e5da542-3f91-4f05-8959-c4c05f9210a9","Type":"ContainerStarted","Data":"eba5bddee02bf1559420b3d28587ba43cbf8f4e226501b94a4b10cbc06527f0e"} Jul 06 19:22:54.290618 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:54.290578 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:22:54.290960 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:54.290649 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:22:55.112032 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:55.111995 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lf8hn" event={"ID":"33a0983c-34cc-4244-99d0-7e3e69f94bbc","Type":"ContainerStarted","Data":"455b1728d081233ff89cfc0c56588decbbe77279901f2366201de16045c3855a"} Jul 06 19:22:55.112244 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:55.112097 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-lf8hn" Jul 06 19:22:55.113572 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:55.113546 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fffdf4997-n76hp" event={"ID":"b354c43c-54c3-4679-9bc3-dea447406712","Type":"ContainerStarted","Data":"419c9980ab4faa03bb687cb0ec3cfd5ed50c97fa6f0c90543617e2638249c571"} Jul 06 19:22:55.115003 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:55.114979 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" event={"ID":"e79c825d-53e2-41e5-b758-eef94b52e1fa","Type":"ContainerStarted","Data":"a61da468599227ee54e52a7cdb4d38376828d855e33d20feb01c115f06255118"} Jul 06 19:22:55.115147 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:55.115125 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:55.116596 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:55.116563 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-d2scm" event={"ID":"0e5da542-3f91-4f05-8959-c4c05f9210a9","Type":"ContainerStarted","Data":"dccce9b66ba57524b94635b484f8f37eb2214f9d42b4ad11be8a7e375a3d68a3"} Jul 06 19:22:55.116694 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:55.116627 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-d2scm" Jul 06 19:22:55.117222 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:55.117199 2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-grwnb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 19:22:55.117345 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:55.117248 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" podUID="e79c825d-53e2-41e5-b758-eef94b52e1fa" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:22:55.169264 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:55.169206 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-d2scm" podStartSLOduration=2.152494468 podStartE2EDuration="6.1691858s" totalImagesPullingTime="4.016691332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:55.167329061 +0000 UTC m=+672.063218593" watchObservedRunningTime="2026-07-06 19:22:55.1691858 +0000 UTC m=+672.065075394" Jul 06 19:22:55.170485 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:55.170452 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-lf8hn" podStartSLOduration=2.083852847 podStartE2EDuration="7.170439667s" totalImagesPullingTime="5.08658682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:55.137426245 +0000 UTC m=+672.033315778" watchObservedRunningTime="2026-07-06 19:22:55.170439667 +0000 UTC m=+672.066329199" Jul 06 19:22:55.193909 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:55.193856 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6fffdf4997-n76hp" podStartSLOduration=1.9598506100000002 podStartE2EDuration="8.193838873s" totalImagesPullingTime="6.233988263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:55.191076626 +0000 UTC m=+672.086966158" watchObservedRunningTime="2026-07-06 19:22:55.193838873 +0000 UTC m=+672.089728442" Jul 06 19:22:55.215614 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:55.215555 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" podStartSLOduration=1.702451896 podStartE2EDuration="6.215534121s" totalImagesPullingTime="4.513082225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:55.214527737 +0000 UTC m=+672.110417270" watchObservedRunningTime="2026-07-06 19:22:55.215534121 +0000 UTC m=+672.111423654" Jul 06 19:22:55.736026 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:55.735989 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:55.839683 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:55.839648 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:56.121885 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.121799 2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-grwnb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 19:22:56.121885 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.121850 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" podUID="e79c825d-53e2-41e5-b758-eef94b52e1fa" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:22:56.234614 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.234576 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr"] Jul 06 19:22:56.238491 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.238459 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.241904 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.241872 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-xfcdt\"" Jul 06 19:22:56.251181 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.251154 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:56.256183 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.256139 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr"] Jul 06 19:22:56.350200 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.350167 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e16a95-6efc-4765-9e90-1a5f17af02e3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.350351 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.350214 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx5tg\" (UniqueName: \"kubernetes.io/projected/a5e16a95-6efc-4765-9e90-1a5f17af02e3-kube-api-access-rx5tg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.350351 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.350275 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e16a95-6efc-4765-9e90-1a5f17af02e3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.350351 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.350321 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e16a95-6efc-4765-9e90-1a5f17af02e3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.350351 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.350346 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e16a95-6efc-4765-9e90-1a5f17af02e3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.350497 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.350365 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e16a95-6efc-4765-9e90-1a5f17af02e3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.350497 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.350393 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e16a95-6efc-4765-9e90-1a5f17af02e3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.350497 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.350412 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e16a95-6efc-4765-9e90-1a5f17af02e3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.350497 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.350432 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e16a95-6efc-4765-9e90-1a5f17af02e3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.403624 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.403537 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:56.451823 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.451791 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e16a95-6efc-4765-9e90-1a5f17af02e3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.451992 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.451830 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e16a95-6efc-4765-9e90-1a5f17af02e3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.451992 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.451848 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e16a95-6efc-4765-9e90-1a5f17af02e3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.451992 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.451942 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e16a95-6efc-4765-9e90-1a5f17af02e3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.451992 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.451979 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e16a95-6efc-4765-9e90-1a5f17af02e3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.452215 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.452021 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx5tg\" (UniqueName: \"kubernetes.io/projected/a5e16a95-6efc-4765-9e90-1a5f17af02e3-kube-api-access-rx5tg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.452215 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.452052 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e16a95-6efc-4765-9e90-1a5f17af02e3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.452215 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.452094 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e16a95-6efc-4765-9e90-1a5f17af02e3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.452366 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.452291 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e16a95-6efc-4765-9e90-1a5f17af02e3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.452366 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.452328 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e16a95-6efc-4765-9e90-1a5f17af02e3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.452523 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.452498 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e16a95-6efc-4765-9e90-1a5f17af02e3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.452601 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.452521 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e16a95-6efc-4765-9e90-1a5f17af02e3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.452713 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.452691 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e16a95-6efc-4765-9e90-1a5f17af02e3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.453074 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.453054 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e16a95-6efc-4765-9e90-1a5f17af02e3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.454329 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.454305 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e16a95-6efc-4765-9e90-1a5f17af02e3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.454507 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.454489 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e16a95-6efc-4765-9e90-1a5f17af02e3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.462309 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.462286 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e16a95-6efc-4765-9e90-1a5f17af02e3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.463663 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.463632 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx5tg\" (UniqueName: \"kubernetes.io/projected/a5e16a95-6efc-4765-9e90-1a5f17af02e3-kube-api-access-rx5tg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rzbhr\" (UID: \"a5e16a95-6efc-4765-9e90-1a5f17af02e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.551820 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.551785 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:22:56.709540 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.709513 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr"] Jul 06 19:22:56.710204 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:22:56.710173 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e16a95_6efc_4765_9e90_1a5f17af02e3.slice/crio-e2e3489e1fd9211adff6a928a6c9ba076e2fb4648dc2cb5fd415ff447c735aec WatchSource:0}: Error finding container e2e3489e1fd9211adff6a928a6c9ba076e2fb4648dc2cb5fd415ff447c735aec: Status 404 returned error can't find the container with id e2e3489e1fd9211adff6a928a6c9ba076e2fb4648dc2cb5fd415ff447c735aec Jul 06 19:22:56.736684 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:56.736657 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:57.126746 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:57.126643 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" event={"ID":"a5e16a95-6efc-4765-9e90-1a5f17af02e3","Type":"ContainerStarted","Data":"e2e3489e1fd9211adff6a928a6c9ba076e2fb4648dc2cb5fd415ff447c735aec"} Jul 06 19:22:58.469359 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.469313 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:58.591513 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.591371 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cq9gn"] Jul 06 19:22:58.598692 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.598658 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xwb8h"] Jul 06 19:22:58.598849 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.598808 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cq9gn" Jul 06 19:22:58.599862 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.599808 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:58.602078 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.601876 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 06 19:22:58.602078 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.601978 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jrvts\"" Jul 06 19:22:58.604034 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.604010 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwb8h" Jul 06 19:22:58.607563 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.606325 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-bsdj8\"" Jul 06 19:22:58.610692 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.607993 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 19:22:58.610692 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.610640 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xwb8h"] Jul 06 19:22:58.616847 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.616827 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cq9gn"] Jul 06 19:22:58.668429 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.668401 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2537d88a-bc4c-46a2-9f91-9620fe11fc1a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cq9gn\" (UID: \"2537d88a-bc4c-46a2-9f91-9620fe11fc1a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cq9gn" Jul 06 19:22:58.668603 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.668467 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5141d320-7665-460c-b7c4-f9e2202470cf-cert\") pod \"kserve-controller-manager-d6f4b65b8-xwb8h\" (UID: \"5141d320-7665-460c-b7c4-f9e2202470cf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwb8h" Jul 06 19:22:58.668603 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.668528 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nrwr\" (UniqueName: \"kubernetes.io/projected/2537d88a-bc4c-46a2-9f91-9620fe11fc1a-kube-api-access-9nrwr\") pod \"llmisvc-controller-manager-777c6bd7d5-cq9gn\" (UID: \"2537d88a-bc4c-46a2-9f91-9620fe11fc1a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cq9gn" Jul 06 19:22:58.668740 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.668608 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw67s\" (UniqueName: \"kubernetes.io/projected/5141d320-7665-460c-b7c4-f9e2202470cf-kube-api-access-mw67s\") pod \"kserve-controller-manager-d6f4b65b8-xwb8h\" (UID: \"5141d320-7665-460c-b7c4-f9e2202470cf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwb8h" Jul 06 19:22:58.737126 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.737033 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:58.770053 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.770025 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw67s\" (UniqueName: \"kubernetes.io/projected/5141d320-7665-460c-b7c4-f9e2202470cf-kube-api-access-mw67s\") pod \"kserve-controller-manager-d6f4b65b8-xwb8h\" (UID: \"5141d320-7665-460c-b7c4-f9e2202470cf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwb8h" Jul 06 19:22:58.770229 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.770156 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2537d88a-bc4c-46a2-9f91-9620fe11fc1a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cq9gn\" (UID: \"2537d88a-bc4c-46a2-9f91-9620fe11fc1a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cq9gn" Jul 06 19:22:58.770229 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.770198 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5141d320-7665-460c-b7c4-f9e2202470cf-cert\") pod \"kserve-controller-manager-d6f4b65b8-xwb8h\" (UID: \"5141d320-7665-460c-b7c4-f9e2202470cf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwb8h" Jul 06 19:22:58.770377 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.770288 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nrwr\" (UniqueName: \"kubernetes.io/projected/2537d88a-bc4c-46a2-9f91-9620fe11fc1a-kube-api-access-9nrwr\") pod \"llmisvc-controller-manager-777c6bd7d5-cq9gn\" (UID: \"2537d88a-bc4c-46a2-9f91-9620fe11fc1a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cq9gn" Jul 06 19:22:58.770456 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:22:58.770317 2566 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 06 19:22:58.770509 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:22:58.770498 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2537d88a-bc4c-46a2-9f91-9620fe11fc1a-cert podName:2537d88a-bc4c-46a2-9f91-9620fe11fc1a nodeName:}" failed. No retries permitted until 2026-07-06 19:22:59.270471998 +0000 UTC m=+676.166361527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2537d88a-bc4c-46a2-9f91-9620fe11fc1a-cert") pod "llmisvc-controller-manager-777c6bd7d5-cq9gn" (UID: "2537d88a-bc4c-46a2-9f91-9620fe11fc1a") : secret "llmisvc-webhook-server-cert" not found Jul 06 19:22:58.773023 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.772999 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5141d320-7665-460c-b7c4-f9e2202470cf-cert\") pod \"kserve-controller-manager-d6f4b65b8-xwb8h\" (UID: \"5141d320-7665-460c-b7c4-f9e2202470cf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwb8h" Jul 06 19:22:58.786670 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.786642 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nrwr\" (UniqueName: \"kubernetes.io/projected/2537d88a-bc4c-46a2-9f91-9620fe11fc1a-kube-api-access-9nrwr\") pod \"llmisvc-controller-manager-777c6bd7d5-cq9gn\" (UID: \"2537d88a-bc4c-46a2-9f91-9620fe11fc1a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cq9gn" Jul 06 19:22:58.786804 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.786650 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw67s\" (UniqueName: \"kubernetes.io/projected/5141d320-7665-460c-b7c4-f9e2202470cf-kube-api-access-mw67s\") pod \"kserve-controller-manager-d6f4b65b8-xwb8h\" (UID: \"5141d320-7665-460c-b7c4-f9e2202470cf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwb8h" Jul 06 19:22:58.924742 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:58.924693 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwb8h" Jul 06 19:22:59.123214 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:59.123131 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:22:59.275290 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:59.275253 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2537d88a-bc4c-46a2-9f91-9620fe11fc1a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cq9gn\" (UID: \"2537d88a-bc4c-46a2-9f91-9620fe11fc1a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cq9gn" Jul 06 19:22:59.278025 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:59.278001 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2537d88a-bc4c-46a2-9f91-9620fe11fc1a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cq9gn\" (UID: \"2537d88a-bc4c-46a2-9f91-9620fe11fc1a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cq9gn" Jul 06 19:22:59.289644 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:59.289615 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:59.516868 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:59.516836 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cq9gn" Jul 06 19:22:59.607895 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:59.607829 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:22:59.608056 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:59.607921 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:22:59.608056 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:59.607963 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:22:59.703255 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:59.703172 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xwb8h"] Jul 06 19:22:59.704652 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:22:59.704621 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5141d320_7665_460c_b7c4_f9e2202470cf.slice/crio-173a04e326c3cd9614459b79db5a3b4439e3d2e88ee5c9abcc1c42740d23c209 WatchSource:0}: Error finding container 173a04e326c3cd9614459b79db5a3b4439e3d2e88ee5c9abcc1c42740d23c209: Status 404 returned error can't find the container with id 173a04e326c3cd9614459b79db5a3b4439e3d2e88ee5c9abcc1c42740d23c209 Jul 06 19:22:59.712273 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:59.712249 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cq9gn"] Jul 06 19:22:59.714048 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:22:59.714027 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2537d88a_bc4c_46a2_9f91_9620fe11fc1a.slice/crio-75337815847a007bd0b8e644e993c214b83cc88b6c23690bf5df74f94f89f189 WatchSource:0}: Error finding container 75337815847a007bd0b8e644e993c214b83cc88b6c23690bf5df74f94f89f189: Status 404 returned error can't find the container with id 75337815847a007bd0b8e644e993c214b83cc88b6c23690bf5df74f94f89f189 Jul 06 19:22:59.724379 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:59.724356 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-grwnb"] Jul 06 19:22:59.724571 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:59.724545 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" podUID="e79c825d-53e2-41e5-b758-eef94b52e1fa" containerName="discovery" containerID="cri-o://a61da468599227ee54e52a7cdb4d38376828d855e33d20feb01c115f06255118" gracePeriod=30 Jul 06 19:22:59.840115 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:59.840084 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:59.963076 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:59.963037 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:59.985620 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:22:59.985594 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:23:00.081055 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.080959 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-csr-ca-configmap\") pod \"e79c825d-53e2-41e5-b758-eef94b52e1fa\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " Jul 06 19:23:00.081274 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.081253 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qwl2\" (UniqueName: \"kubernetes.io/projected/e79c825d-53e2-41e5-b758-eef94b52e1fa-kube-api-access-5qwl2\") pod \"e79c825d-53e2-41e5-b758-eef94b52e1fa\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " Jul 06 19:23:00.081435 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.081413 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "e79c825d-53e2-41e5-b758-eef94b52e1fa" (UID: "e79c825d-53e2-41e5-b758-eef94b52e1fa"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:23:00.081511 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.081420 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e79c825d-53e2-41e5-b758-eef94b52e1fa-local-certs\") pod \"e79c825d-53e2-41e5-b758-eef94b52e1fa\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " Jul 06 19:23:00.081571 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.081513 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-kubeconfig\") pod \"e79c825d-53e2-41e5-b758-eef94b52e1fa\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " Jul 06 19:23:00.081571 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.081544 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e79c825d-53e2-41e5-b758-eef94b52e1fa-cacerts\") pod \"e79c825d-53e2-41e5-b758-eef94b52e1fa\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " Jul 06 19:23:00.081663 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.081621 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-csr-dns-cert\") pod \"e79c825d-53e2-41e5-b758-eef94b52e1fa\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " Jul 06 19:23:00.081663 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.081657 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-token\") pod \"e79c825d-53e2-41e5-b758-eef94b52e1fa\" (UID: \"e79c825d-53e2-41e5-b758-eef94b52e1fa\") " Jul 06 19:23:00.082179 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.081918 2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-csr-ca-configmap\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:23:00.085329 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.084943 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e79c825d-53e2-41e5-b758-eef94b52e1fa-cacerts" (OuterVolumeSpecName: "cacerts") pod "e79c825d-53e2-41e5-b758-eef94b52e1fa" (UID: "e79c825d-53e2-41e5-b758-eef94b52e1fa"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:23:00.085329 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.085007 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79c825d-53e2-41e5-b758-eef94b52e1fa-kube-api-access-5qwl2" (OuterVolumeSpecName: "kube-api-access-5qwl2") pod "e79c825d-53e2-41e5-b758-eef94b52e1fa" (UID: "e79c825d-53e2-41e5-b758-eef94b52e1fa"). InnerVolumeSpecName "kube-api-access-5qwl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:23:00.085329 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.085182 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "e79c825d-53e2-41e5-b758-eef94b52e1fa" (UID: "e79c825d-53e2-41e5-b758-eef94b52e1fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:23:00.085329 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.085290 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "e79c825d-53e2-41e5-b758-eef94b52e1fa" (UID: "e79c825d-53e2-41e5-b758-eef94b52e1fa"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:23:00.085619 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.085485 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79c825d-53e2-41e5-b758-eef94b52e1fa-local-certs" (OuterVolumeSpecName: "local-certs") pod "e79c825d-53e2-41e5-b758-eef94b52e1fa" (UID: "e79c825d-53e2-41e5-b758-eef94b52e1fa"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:23:00.086528 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.086496 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "e79c825d-53e2-41e5-b758-eef94b52e1fa" (UID: "e79c825d-53e2-41e5-b758-eef94b52e1fa"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:23:00.094695 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.094667 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:00.142936 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.142865 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwb8h" event={"ID":"5141d320-7665-460c-b7c4-f9e2202470cf","Type":"ContainerStarted","Data":"173a04e326c3cd9614459b79db5a3b4439e3d2e88ee5c9abcc1c42740d23c209"} Jul 06 19:23:00.144610 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.144574 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" event={"ID":"a5e16a95-6efc-4765-9e90-1a5f17af02e3","Type":"ContainerStarted","Data":"79f5efdacfec8b9e8bca0774a35341d8c5fd9cafb85511653ac30094a7bc0966"} Jul 06 19:23:00.145763 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.145741 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cq9gn" event={"ID":"2537d88a-bc4c-46a2-9f91-9620fe11fc1a","Type":"ContainerStarted","Data":"75337815847a007bd0b8e644e993c214b83cc88b6c23690bf5df74f94f89f189"} Jul 06 19:23:00.146950 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.146930 2566 generic.go:356] "Generic (PLEG): container finished" podID="e79c825d-53e2-41e5-b758-eef94b52e1fa" containerID="a61da468599227ee54e52a7cdb4d38376828d855e33d20feb01c115f06255118" exitCode=0 Jul 06 19:23:00.147035 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.146960 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" event={"ID":"e79c825d-53e2-41e5-b758-eef94b52e1fa","Type":"ContainerDied","Data":"a61da468599227ee54e52a7cdb4d38376828d855e33d20feb01c115f06255118"} Jul 06 19:23:00.147035 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.146967 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" Jul 06 19:23:00.147035 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.146985 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-grwnb" event={"ID":"e79c825d-53e2-41e5-b758-eef94b52e1fa","Type":"ContainerDied","Data":"468a41ba4954100788590821fe526e592f8551f613adb7c4b3f27c1255e5b8a0"} Jul 06 19:23:00.147035 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.147004 2566 scope.go:117] "RemoveContainer" containerID="a61da468599227ee54e52a7cdb4d38376828d855e33d20feb01c115f06255118" Jul 06 19:23:00.156251 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.156232 2566 scope.go:117] "RemoveContainer" containerID="a61da468599227ee54e52a7cdb4d38376828d855e33d20feb01c115f06255118" Jul 06 19:23:00.156503 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:23:00.156480 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61da468599227ee54e52a7cdb4d38376828d855e33d20feb01c115f06255118\": container with ID starting with a61da468599227ee54e52a7cdb4d38376828d855e33d20feb01c115f06255118 not found: ID does not exist" containerID="a61da468599227ee54e52a7cdb4d38376828d855e33d20feb01c115f06255118" Jul 06 19:23:00.156565 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.156510 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61da468599227ee54e52a7cdb4d38376828d855e33d20feb01c115f06255118"} err="failed to get container status \"a61da468599227ee54e52a7cdb4d38376828d855e33d20feb01c115f06255118\": rpc error: code = NotFound desc = could not find container \"a61da468599227ee54e52a7cdb4d38376828d855e33d20feb01c115f06255118\": container with ID starting with a61da468599227ee54e52a7cdb4d38376828d855e33d20feb01c115f06255118 not found: ID does not exist" Jul 06 19:23:00.182527 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.182499 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qwl2\" (UniqueName: \"kubernetes.io/projected/e79c825d-53e2-41e5-b758-eef94b52e1fa-kube-api-access-5qwl2\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:23:00.182527 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.182522 2566 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e79c825d-53e2-41e5-b758-eef94b52e1fa-local-certs\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:23:00.182527 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.182533 2566 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-kubeconfig\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:23:00.182709 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.182542 2566 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e79c825d-53e2-41e5-b758-eef94b52e1fa-cacerts\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:23:00.182709 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.182551 2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-csr-dns-cert\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:23:00.182709 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.182559 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e79c825d-53e2-41e5-b758-eef94b52e1fa-istio-token\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:23:00.192578 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.192538 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" podStartSLOduration=1.297302248 podStartE2EDuration="4.192522651s" totalImagesPullingTime="2.895220403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:00.188008578 +0000 UTC m=+677.083898110" watchObservedRunningTime="2026-07-06 19:23:00.192522651 +0000 UTC m=+677.088412182" Jul 06 19:23:00.217188 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.217153 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-grwnb"] Jul 06 19:23:00.223123 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.223093 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-grwnb"] Jul 06 19:23:00.552224 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.552185 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:23:00.557651 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:00.557618 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:23:01.153762 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:01.153705 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:23:01.154749 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:01.154704 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rzbhr" Jul 06 19:23:01.763059 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:01.763030 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e79c825d-53e2-41e5-b758-eef94b52e1fa" path="/var/lib/kubelet/pods/e79c825d-53e2-41e5-b758-eef94b52e1fa/volumes" Jul 06 19:23:04.174263 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:04.174224 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cq9gn" event={"ID":"2537d88a-bc4c-46a2-9f91-9620fe11fc1a","Type":"ContainerStarted","Data":"fd36e82cbcf8ef8312858ae0327aea0bd45545bb1b0b9ce4977193a6bf434e45"} Jul 06 19:23:04.174716 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:04.174325 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cq9gn" Jul 06 19:23:04.175836 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:04.175813 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwb8h" event={"ID":"5141d320-7665-460c-b7c4-f9e2202470cf","Type":"ContainerStarted","Data":"6a2afe272b35c18dec42e040381b321206339b411e20aae0fcd87c2049847101"} Jul 06 19:23:04.175948 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:04.175937 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwb8h" Jul 06 19:23:04.191905 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:04.191858 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cq9gn" podStartSLOduration=2.428573815 podStartE2EDuration="6.191845058s" totalImagesPullingTime="3.763271243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:04.190328508 +0000 UTC m=+681.086218040" watchObservedRunningTime="2026-07-06 19:23:04.191845058 +0000 UTC m=+681.087734590" Jul 06 19:23:06.123888 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:06.123854 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-lf8hn" Jul 06 19:23:06.125871 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:06.125848 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-d2scm" Jul 06 19:23:06.142201 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:06.142154 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwb8h" podStartSLOduration=4.368036906 podStartE2EDuration="8.142137375s" totalImagesPullingTime="3.774100469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:04.211467528 +0000 UTC m=+681.107357071" watchObservedRunningTime="2026-07-06 19:23:06.142137375 +0000 UTC m=+683.038026907" Jul 06 19:23:15.458143 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:15.458093 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:15.513406 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:15.513351 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:15.602442 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:15.602392 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:15.702792 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:15.702750 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:16.195794 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:16.195708 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:25.962307 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:25.962264 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:26.029561 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:26.029528 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:26.102756 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:26.102699 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:35.182625 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:35.182531 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cq9gn" Jul 06 19:23:35.185514 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:35.185494 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwb8h" Jul 06 19:23:48.936005 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:48.935962 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:49.033985 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.033950 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:49.051889 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.051859 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sx8qt"] Jul 06 19:23:49.052167 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.052156 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e79c825d-53e2-41e5-b758-eef94b52e1fa" containerName="discovery" Jul 06 19:23:49.052216 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.052168 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e79c825d-53e2-41e5-b758-eef94b52e1fa" containerName="discovery" Jul 06 19:23:49.052249 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.052223 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e79c825d-53e2-41e5-b758-eef94b52e1fa" containerName="discovery" Jul 06 19:23:49.056184 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.056164 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sx8qt" Jul 06 19:23:49.058816 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.058794 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:23:49.059634 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.059611 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-zg9dq\"" Jul 06 19:23:49.059634 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.059624 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:23:49.064707 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.064678 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sx8qt"] Jul 06 19:23:49.097096 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.097063 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfvvf\" (UniqueName: \"kubernetes.io/projected/d210e12d-5e37-4437-903a-963cdcdcaab5-kube-api-access-zfvvf\") pod \"kuadrant-operator-catalog-sx8qt\" (UID: \"d210e12d-5e37-4437-903a-963cdcdcaab5\") " pod="kuadrant-system/kuadrant-operator-catalog-sx8qt" Jul 06 19:23:49.134668 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.134636 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:49.197964 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.197884 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvvf\" (UniqueName: \"kubernetes.io/projected/d210e12d-5e37-4437-903a-963cdcdcaab5-kube-api-access-zfvvf\") pod \"kuadrant-operator-catalog-sx8qt\" (UID: \"d210e12d-5e37-4437-903a-963cdcdcaab5\") " pod="kuadrant-system/kuadrant-operator-catalog-sx8qt" Jul 06 19:23:49.205713 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.205686 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfvvf\" (UniqueName: \"kubernetes.io/projected/d210e12d-5e37-4437-903a-963cdcdcaab5-kube-api-access-zfvvf\") pod \"kuadrant-operator-catalog-sx8qt\" (UID: \"d210e12d-5e37-4437-903a-963cdcdcaab5\") " pod="kuadrant-system/kuadrant-operator-catalog-sx8qt" Jul 06 19:23:49.367875 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.367845 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sx8qt" Jul 06 19:23:49.419046 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.419015 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sx8qt"] Jul 06 19:23:49.498376 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.498310 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sx8qt"] Jul 06 19:23:49.500864 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:23:49.500833 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd210e12d_5e37_4437_903a_963cdcdcaab5.slice/crio-49ef7f76e220c1214caca68ca46296e6ec557805aa995a76b7963d11504f9bb8 WatchSource:0}: Error finding container 49ef7f76e220c1214caca68ca46296e6ec557805aa995a76b7963d11504f9bb8: Status 404 returned error can't find the container with id 49ef7f76e220c1214caca68ca46296e6ec557805aa995a76b7963d11504f9bb8 Jul 06 19:23:49.632395 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.632357 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-svpfq"] Jul 06 19:23:49.636786 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.636762 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-svpfq" Jul 06 19:23:49.646779 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.646752 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-svpfq"] Jul 06 19:23:49.701750 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.701650 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9lsj\" (UniqueName: \"kubernetes.io/projected/91d7dfa4-0892-4cac-bbeb-34565f1c8540-kube-api-access-s9lsj\") pod \"kuadrant-operator-catalog-svpfq\" (UID: \"91d7dfa4-0892-4cac-bbeb-34565f1c8540\") " pod="kuadrant-system/kuadrant-operator-catalog-svpfq" Jul 06 19:23:49.802683 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.802647 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9lsj\" (UniqueName: \"kubernetes.io/projected/91d7dfa4-0892-4cac-bbeb-34565f1c8540-kube-api-access-s9lsj\") pod \"kuadrant-operator-catalog-svpfq\" (UID: \"91d7dfa4-0892-4cac-bbeb-34565f1c8540\") " pod="kuadrant-system/kuadrant-operator-catalog-svpfq" Jul 06 19:23:49.811844 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.811819 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9lsj\" (UniqueName: \"kubernetes.io/projected/91d7dfa4-0892-4cac-bbeb-34565f1c8540-kube-api-access-s9lsj\") pod \"kuadrant-operator-catalog-svpfq\" (UID: \"91d7dfa4-0892-4cac-bbeb-34565f1c8540\") " pod="kuadrant-system/kuadrant-operator-catalog-svpfq" Jul 06 19:23:49.864763 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.864732 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:49.946407 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:49.946373 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-svpfq" Jul 06 19:23:50.056839 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:50.056775 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:50.083339 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:50.083158 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-svpfq"] Jul 06 19:23:50.085380 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:23:50.085355 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d7dfa4_0892_4cac_bbeb_34565f1c8540.slice/crio-c43fd5681cc5ec0d968c8fef7576c0608af0f5f9b052df950c03d923d6b8902a WatchSource:0}: Error finding container c43fd5681cc5ec0d968c8fef7576c0608af0f5f9b052df950c03d923d6b8902a: Status 404 returned error can't find the container with id c43fd5681cc5ec0d968c8fef7576c0608af0f5f9b052df950c03d923d6b8902a Jul 06 19:23:50.375529 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:50.375454 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sx8qt" event={"ID":"d210e12d-5e37-4437-903a-963cdcdcaab5","Type":"ContainerStarted","Data":"49ef7f76e220c1214caca68ca46296e6ec557805aa995a76b7963d11504f9bb8"} Jul 06 19:23:50.376685 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:50.376663 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-svpfq" event={"ID":"91d7dfa4-0892-4cac-bbeb-34565f1c8540","Type":"ContainerStarted","Data":"c43fd5681cc5ec0d968c8fef7576c0608af0f5f9b052df950c03d923d6b8902a"} Jul 06 19:23:52.387562 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:52.387522 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sx8qt" event={"ID":"d210e12d-5e37-4437-903a-963cdcdcaab5","Type":"ContainerStarted","Data":"c76810fea26e1f5fdd2203342d6f8a7f3e268bff3061baf1ac51821c43e91de3"} Jul 06 19:23:52.387562 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:52.387544 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-sx8qt" podUID="d210e12d-5e37-4437-903a-963cdcdcaab5" containerName="registry-server" containerID="cri-o://c76810fea26e1f5fdd2203342d6f8a7f3e268bff3061baf1ac51821c43e91de3" gracePeriod=2 Jul 06 19:23:52.389109 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:52.389085 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-svpfq" event={"ID":"91d7dfa4-0892-4cac-bbeb-34565f1c8540","Type":"ContainerStarted","Data":"7bf68a91991179e18751d7429e9358d5d1e2b9eb7763551ff7590ef278f40630"} Jul 06 19:23:52.402942 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:52.402896 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-sx8qt" podStartSLOduration=1.180969768 podStartE2EDuration="3.402877623s" totalImagesPullingTime="2.221907855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:52.402426558 +0000 UTC m=+729.298316092" watchObservedRunningTime="2026-07-06 19:23:52.402877623 +0000 UTC m=+729.298767158" Jul 06 19:23:52.418612 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:52.418568 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-svpfq" podStartSLOduration=1.780249243 podStartE2EDuration="3.418554725s" totalImagesPullingTime="1.638305482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:52.417231014 +0000 UTC m=+729.313120545" watchObservedRunningTime="2026-07-06 19:23:52.418554725 +0000 UTC m=+729.314444257" Jul 06 19:23:52.629179 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:52.629155 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sx8qt" Jul 06 19:23:52.728042 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:52.727978 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfvvf\" (UniqueName: \"kubernetes.io/projected/d210e12d-5e37-4437-903a-963cdcdcaab5-kube-api-access-zfvvf\") pod \"d210e12d-5e37-4437-903a-963cdcdcaab5\" (UID: \"d210e12d-5e37-4437-903a-963cdcdcaab5\") " Jul 06 19:23:52.730166 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:52.730142 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d210e12d-5e37-4437-903a-963cdcdcaab5-kube-api-access-zfvvf" (OuterVolumeSpecName: "kube-api-access-zfvvf") pod "d210e12d-5e37-4437-903a-963cdcdcaab5" (UID: "d210e12d-5e37-4437-903a-963cdcdcaab5"). InnerVolumeSpecName "kube-api-access-zfvvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:23:52.829274 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:52.829226 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfvvf\" (UniqueName: \"kubernetes.io/projected/d210e12d-5e37-4437-903a-963cdcdcaab5-kube-api-access-zfvvf\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:23:53.395007 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:53.394973 2566 generic.go:356] "Generic (PLEG): container finished" podID="d210e12d-5e37-4437-903a-963cdcdcaab5" containerID="c76810fea26e1f5fdd2203342d6f8a7f3e268bff3061baf1ac51821c43e91de3" exitCode=0 Jul 06 19:23:53.395502 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:53.395016 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sx8qt" Jul 06 19:23:53.395502 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:53.395067 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sx8qt" event={"ID":"d210e12d-5e37-4437-903a-963cdcdcaab5","Type":"ContainerDied","Data":"c76810fea26e1f5fdd2203342d6f8a7f3e268bff3061baf1ac51821c43e91de3"} Jul 06 19:23:53.395502 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:53.395115 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sx8qt" event={"ID":"d210e12d-5e37-4437-903a-963cdcdcaab5","Type":"ContainerDied","Data":"49ef7f76e220c1214caca68ca46296e6ec557805aa995a76b7963d11504f9bb8"} Jul 06 19:23:53.395502 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:53.395136 2566 scope.go:117] "RemoveContainer" containerID="c76810fea26e1f5fdd2203342d6f8a7f3e268bff3061baf1ac51821c43e91de3" Jul 06 19:23:53.403823 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:53.403807 2566 scope.go:117] "RemoveContainer" containerID="c76810fea26e1f5fdd2203342d6f8a7f3e268bff3061baf1ac51821c43e91de3" Jul 06 19:23:53.404065 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:23:53.404049 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c76810fea26e1f5fdd2203342d6f8a7f3e268bff3061baf1ac51821c43e91de3\": container with ID starting with c76810fea26e1f5fdd2203342d6f8a7f3e268bff3061baf1ac51821c43e91de3 not found: ID does not exist" containerID="c76810fea26e1f5fdd2203342d6f8a7f3e268bff3061baf1ac51821c43e91de3" Jul 06 19:23:53.404122 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:53.404075 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c76810fea26e1f5fdd2203342d6f8a7f3e268bff3061baf1ac51821c43e91de3"} err="failed to get container status \"c76810fea26e1f5fdd2203342d6f8a7f3e268bff3061baf1ac51821c43e91de3\": rpc error: code = NotFound desc = could not find container \"c76810fea26e1f5fdd2203342d6f8a7f3e268bff3061baf1ac51821c43e91de3\": container with ID starting with c76810fea26e1f5fdd2203342d6f8a7f3e268bff3061baf1ac51821c43e91de3 not found: ID does not exist" Jul 06 19:23:53.416027 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:53.416005 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sx8qt"] Jul 06 19:23:53.420174 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:53.420152 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sx8qt"] Jul 06 19:23:53.763202 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:53.763170 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d210e12d-5e37-4437-903a-963cdcdcaab5" path="/var/lib/kubelet/pods/d210e12d-5e37-4437-903a-963cdcdcaab5/volumes" Jul 06 19:23:59.947328 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:59.947292 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-svpfq" Jul 06 19:23:59.947791 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:59.947352 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-svpfq" Jul 06 19:23:59.970371 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:23:59.970343 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-svpfq" Jul 06 19:24:00.451823 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:00.451796 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-svpfq" Jul 06 19:24:04.864580 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:04.864542 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn"] Jul 06 19:24:04.865127 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:04.865058 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d210e12d-5e37-4437-903a-963cdcdcaab5" containerName="registry-server" Jul 06 19:24:04.865127 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:04.865076 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d210e12d-5e37-4437-903a-963cdcdcaab5" containerName="registry-server" Jul 06 19:24:04.865241 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:04.865150 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d210e12d-5e37-4437-903a-963cdcdcaab5" containerName="registry-server" Jul 06 19:24:04.869694 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:04.869673 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn" Jul 06 19:24:04.872082 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:04.872064 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jfkvd\"" Jul 06 19:24:04.875879 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:04.875858 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn"] Jul 06 19:24:04.930709 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:04.930670 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab2c8e51-541d-4f44-9d87-a64dd000f2d7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn\" (UID: \"ab2c8e51-541d-4f44-9d87-a64dd000f2d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn" Jul 06 19:24:04.930926 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:04.930749 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btwz9\" (UniqueName: \"kubernetes.io/projected/ab2c8e51-541d-4f44-9d87-a64dd000f2d7-kube-api-access-btwz9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn\" (UID: \"ab2c8e51-541d-4f44-9d87-a64dd000f2d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn" Jul 06 19:24:04.930926 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:04.930874 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab2c8e51-541d-4f44-9d87-a64dd000f2d7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn\" (UID: \"ab2c8e51-541d-4f44-9d87-a64dd000f2d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn" Jul 06 19:24:05.032329 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.032291 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab2c8e51-541d-4f44-9d87-a64dd000f2d7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn\" (UID: \"ab2c8e51-541d-4f44-9d87-a64dd000f2d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn" Jul 06 19:24:05.032568 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.032343 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btwz9\" (UniqueName: \"kubernetes.io/projected/ab2c8e51-541d-4f44-9d87-a64dd000f2d7-kube-api-access-btwz9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn\" (UID: \"ab2c8e51-541d-4f44-9d87-a64dd000f2d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn" Jul 06 19:24:05.032568 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.032404 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab2c8e51-541d-4f44-9d87-a64dd000f2d7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn\" (UID: \"ab2c8e51-541d-4f44-9d87-a64dd000f2d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn" Jul 06 19:24:05.032809 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.032771 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab2c8e51-541d-4f44-9d87-a64dd000f2d7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn\" (UID: \"ab2c8e51-541d-4f44-9d87-a64dd000f2d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn" Jul 06 19:24:05.032895 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.032819 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab2c8e51-541d-4f44-9d87-a64dd000f2d7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn\" (UID: \"ab2c8e51-541d-4f44-9d87-a64dd000f2d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn" Jul 06 19:24:05.041534 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.041512 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btwz9\" (UniqueName: \"kubernetes.io/projected/ab2c8e51-541d-4f44-9d87-a64dd000f2d7-kube-api-access-btwz9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn\" (UID: \"ab2c8e51-541d-4f44-9d87-a64dd000f2d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn" Jul 06 19:24:05.179858 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.179830 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn" Jul 06 19:24:05.317274 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.317248 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn"] Jul 06 19:24:05.319671 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:24:05.319646 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab2c8e51_541d_4f44_9d87_a64dd000f2d7.slice/crio-637dfa417e82aeaee31795ddec898264cbe01588089e0b5b54d0ae4cf217f9a0 WatchSource:0}: Error finding container 637dfa417e82aeaee31795ddec898264cbe01588089e0b5b54d0ae4cf217f9a0: Status 404 returned error can't find the container with id 637dfa417e82aeaee31795ddec898264cbe01588089e0b5b54d0ae4cf217f9a0 Jul 06 19:24:05.453694 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.453602 2566 generic.go:356] "Generic (PLEG): container finished" podID="ab2c8e51-541d-4f44-9d87-a64dd000f2d7" containerID="ee477e6e9e6b2cf33c05a98bae29a121465eff2f001f5b000809752b169b844c" exitCode=0 Jul 06 19:24:05.453694 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.453682 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn" event={"ID":"ab2c8e51-541d-4f44-9d87-a64dd000f2d7","Type":"ContainerDied","Data":"ee477e6e9e6b2cf33c05a98bae29a121465eff2f001f5b000809752b169b844c"} Jul 06 19:24:05.453993 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.453781 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn" event={"ID":"ab2c8e51-541d-4f44-9d87-a64dd000f2d7","Type":"ContainerStarted","Data":"637dfa417e82aeaee31795ddec898264cbe01588089e0b5b54d0ae4cf217f9a0"} Jul 06 19:24:05.466916 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.466890 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm"] Jul 06 19:24:05.470297 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.470283 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm" Jul 06 19:24:05.482270 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.482243 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm"] Jul 06 19:24:05.537508 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.537469 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da1a9d25-a8bf-4d2b-9531-c15710aa8192-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm\" (UID: \"da1a9d25-a8bf-4d2b-9531-c15710aa8192\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm" Jul 06 19:24:05.537664 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.537518 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfppf\" (UniqueName: \"kubernetes.io/projected/da1a9d25-a8bf-4d2b-9531-c15710aa8192-kube-api-access-zfppf\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm\" (UID: \"da1a9d25-a8bf-4d2b-9531-c15710aa8192\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm" Jul 06 19:24:05.537664 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.537560 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da1a9d25-a8bf-4d2b-9531-c15710aa8192-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm\" (UID: \"da1a9d25-a8bf-4d2b-9531-c15710aa8192\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm" Jul 06 19:24:05.638430 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.638372 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da1a9d25-a8bf-4d2b-9531-c15710aa8192-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm\" (UID: \"da1a9d25-a8bf-4d2b-9531-c15710aa8192\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm" Jul 06 19:24:05.638430 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.638436 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfppf\" (UniqueName: \"kubernetes.io/projected/da1a9d25-a8bf-4d2b-9531-c15710aa8192-kube-api-access-zfppf\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm\" (UID: \"da1a9d25-a8bf-4d2b-9531-c15710aa8192\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm" Jul 06 19:24:05.638690 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.638542 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da1a9d25-a8bf-4d2b-9531-c15710aa8192-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm\" (UID: \"da1a9d25-a8bf-4d2b-9531-c15710aa8192\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm" Jul 06 19:24:05.638832 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.638810 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da1a9d25-a8bf-4d2b-9531-c15710aa8192-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm\" (UID: \"da1a9d25-a8bf-4d2b-9531-c15710aa8192\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm" Jul 06 19:24:05.638941 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.638922 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da1a9d25-a8bf-4d2b-9531-c15710aa8192-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm\" (UID: \"da1a9d25-a8bf-4d2b-9531-c15710aa8192\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm" Jul 06 19:24:05.647577 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.647552 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfppf\" (UniqueName: \"kubernetes.io/projected/da1a9d25-a8bf-4d2b-9531-c15710aa8192-kube-api-access-zfppf\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm\" (UID: \"da1a9d25-a8bf-4d2b-9531-c15710aa8192\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm" Jul 06 19:24:05.788985 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.788900 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm" Jul 06 19:24:05.879028 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.878998 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85"] Jul 06 19:24:05.883966 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.883940 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85" Jul 06 19:24:05.894064 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.893242 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85"] Jul 06 19:24:05.926281 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:05.926225 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm"] Jul 06 19:24:05.928308 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:24:05.928288 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1a9d25_a8bf_4d2b_9531_c15710aa8192.slice/crio-beb39865327de43dad21bd4b6f3f1a41c7edc5f4bbd09ebb6b394e8be1414ade WatchSource:0}: Error finding container beb39865327de43dad21bd4b6f3f1a41c7edc5f4bbd09ebb6b394e8be1414ade: Status 404 returned error can't find the container with id beb39865327de43dad21bd4b6f3f1a41c7edc5f4bbd09ebb6b394e8be1414ade Jul 06 19:24:06.043597 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.043567 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90fd6204-edb5-4cce-aee2-c576f74414e2-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85\" (UID: \"90fd6204-edb5-4cce-aee2-c576f74414e2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85" Jul 06 19:24:06.043745 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.043622 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztscf\" (UniqueName: \"kubernetes.io/projected/90fd6204-edb5-4cce-aee2-c576f74414e2-kube-api-access-ztscf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85\" (UID: \"90fd6204-edb5-4cce-aee2-c576f74414e2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85" Jul 06 19:24:06.043811 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.043748 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90fd6204-edb5-4cce-aee2-c576f74414e2-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85\" (UID: \"90fd6204-edb5-4cce-aee2-c576f74414e2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85" Jul 06 19:24:06.144144 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.144112 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90fd6204-edb5-4cce-aee2-c576f74414e2-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85\" (UID: \"90fd6204-edb5-4cce-aee2-c576f74414e2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85" Jul 06 19:24:06.144274 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.144159 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90fd6204-edb5-4cce-aee2-c576f74414e2-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85\" (UID: \"90fd6204-edb5-4cce-aee2-c576f74414e2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85" Jul 06 19:24:06.144274 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.144197 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztscf\" (UniqueName: \"kubernetes.io/projected/90fd6204-edb5-4cce-aee2-c576f74414e2-kube-api-access-ztscf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85\" (UID: \"90fd6204-edb5-4cce-aee2-c576f74414e2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85" Jul 06 19:24:06.144529 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.144505 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90fd6204-edb5-4cce-aee2-c576f74414e2-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85\" (UID: \"90fd6204-edb5-4cce-aee2-c576f74414e2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85" Jul 06 19:24:06.144571 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.144516 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90fd6204-edb5-4cce-aee2-c576f74414e2-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85\" (UID: \"90fd6204-edb5-4cce-aee2-c576f74414e2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85" Jul 06 19:24:06.153583 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.153557 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztscf\" (UniqueName: \"kubernetes.io/projected/90fd6204-edb5-4cce-aee2-c576f74414e2-kube-api-access-ztscf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85\" (UID: \"90fd6204-edb5-4cce-aee2-c576f74414e2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85" Jul 06 19:24:06.196797 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.196769 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85" Jul 06 19:24:06.270792 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.270763 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4"] Jul 06 19:24:06.274394 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.274375 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4" Jul 06 19:24:06.281855 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.281801 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4"] Jul 06 19:24:06.343384 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.343320 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85"] Jul 06 19:24:06.345716 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.345697 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr55z\" (UniqueName: \"kubernetes.io/projected/a62f42d5-f4f7-48bf-ad8b-e87e673bac72-kube-api-access-xr55z\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4\" (UID: \"a62f42d5-f4f7-48bf-ad8b-e87e673bac72\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4" Jul 06 19:24:06.345775 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.345749 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a62f42d5-f4f7-48bf-ad8b-e87e673bac72-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4\" (UID: \"a62f42d5-f4f7-48bf-ad8b-e87e673bac72\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4" Jul 06 19:24:06.345813 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.345773 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a62f42d5-f4f7-48bf-ad8b-e87e673bac72-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4\" (UID: \"a62f42d5-f4f7-48bf-ad8b-e87e673bac72\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4" Jul 06 19:24:06.357599 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:24:06.357568 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fd6204_edb5_4cce_aee2_c576f74414e2.slice/crio-6b573fd16cde30684cd74a6e69ace1148b46d70a30830dffb1dd933ad2c45ac3 WatchSource:0}: Error finding container 6b573fd16cde30684cd74a6e69ace1148b46d70a30830dffb1dd933ad2c45ac3: Status 404 returned error can't find the container with id 6b573fd16cde30684cd74a6e69ace1148b46d70a30830dffb1dd933ad2c45ac3 Jul 06 19:24:06.446560 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.446529 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr55z\" (UniqueName: \"kubernetes.io/projected/a62f42d5-f4f7-48bf-ad8b-e87e673bac72-kube-api-access-xr55z\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4\" (UID: \"a62f42d5-f4f7-48bf-ad8b-e87e673bac72\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4" Jul 06 19:24:06.446560 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.446566 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a62f42d5-f4f7-48bf-ad8b-e87e673bac72-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4\" (UID: \"a62f42d5-f4f7-48bf-ad8b-e87e673bac72\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4" Jul 06 19:24:06.446816 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.446585 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a62f42d5-f4f7-48bf-ad8b-e87e673bac72-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4\" (UID: \"a62f42d5-f4f7-48bf-ad8b-e87e673bac72\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4" Jul 06 19:24:06.446944 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.446927 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a62f42d5-f4f7-48bf-ad8b-e87e673bac72-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4\" (UID: \"a62f42d5-f4f7-48bf-ad8b-e87e673bac72\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4" Jul 06 19:24:06.447012 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.446963 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a62f42d5-f4f7-48bf-ad8b-e87e673bac72-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4\" (UID: \"a62f42d5-f4f7-48bf-ad8b-e87e673bac72\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4" Jul 06 19:24:06.456030 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.456008 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr55z\" (UniqueName: \"kubernetes.io/projected/a62f42d5-f4f7-48bf-ad8b-e87e673bac72-kube-api-access-xr55z\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4\" (UID: \"a62f42d5-f4f7-48bf-ad8b-e87e673bac72\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4" Jul 06 19:24:06.460774 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.460750 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85" event={"ID":"90fd6204-edb5-4cce-aee2-c576f74414e2","Type":"ContainerStarted","Data":"ba8474df0fe7e60c5f8a269cc0eb3de8cf9b7710f3e6522bffe3bdd4f538d368"} Jul 06 19:24:06.460898 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.460794 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85" event={"ID":"90fd6204-edb5-4cce-aee2-c576f74414e2","Type":"ContainerStarted","Data":"6b573fd16cde30684cd74a6e69ace1148b46d70a30830dffb1dd933ad2c45ac3"} Jul 06 19:24:06.462287 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.462261 2566 generic.go:356] "Generic (PLEG): container finished" podID="da1a9d25-a8bf-4d2b-9531-c15710aa8192" containerID="19bbe17cd510bf2471651635ce1e591f543570cb771c3455351237ad66fe3265" exitCode=0 Jul 06 19:24:06.462382 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.462339 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm" event={"ID":"da1a9d25-a8bf-4d2b-9531-c15710aa8192","Type":"ContainerDied","Data":"19bbe17cd510bf2471651635ce1e591f543570cb771c3455351237ad66fe3265"} Jul 06 19:24:06.462382 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.462380 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm" event={"ID":"da1a9d25-a8bf-4d2b-9531-c15710aa8192","Type":"ContainerStarted","Data":"beb39865327de43dad21bd4b6f3f1a41c7edc5f4bbd09ebb6b394e8be1414ade"} Jul 06 19:24:06.464103 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.464083 2566 generic.go:356] "Generic (PLEG): container finished" podID="ab2c8e51-541d-4f44-9d87-a64dd000f2d7" containerID="289e0e478b9cba48b9d6ce9927f8baab007c5b07ec9c1a1c74d8dcf565f78b6b" exitCode=0 Jul 06 19:24:06.464199 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.464113 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn" event={"ID":"ab2c8e51-541d-4f44-9d87-a64dd000f2d7","Type":"ContainerDied","Data":"289e0e478b9cba48b9d6ce9927f8baab007c5b07ec9c1a1c74d8dcf565f78b6b"} Jul 06 19:24:06.586371 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.586294 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4" Jul 06 19:24:06.719293 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:06.719268 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4"] Jul 06 19:24:06.721520 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:24:06.721492 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62f42d5_f4f7_48bf_ad8b_e87e673bac72.slice/crio-6b610b8417fb2d3c6c646837bdc21306c8554d11dbb91d7755d554e94167d746 WatchSource:0}: Error finding container 6b610b8417fb2d3c6c646837bdc21306c8554d11dbb91d7755d554e94167d746: Status 404 returned error can't find the container with id 6b610b8417fb2d3c6c646837bdc21306c8554d11dbb91d7755d554e94167d746 Jul 06 19:24:07.471256 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:07.471227 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm" event={"ID":"da1a9d25-a8bf-4d2b-9531-c15710aa8192","Type":"ContainerStarted","Data":"a17e292e6d101f66165281bb5d06527e1abc0b8f996004097ec962590677ebc2"} Jul 06 19:24:07.472613 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:07.472592 2566 generic.go:356] "Generic (PLEG): container finished" podID="a62f42d5-f4f7-48bf-ad8b-e87e673bac72" containerID="c766e442890a3078dcfbc1564b9d94cdbbe07d89dfa5a8a51757fddf443c81ab" exitCode=0 Jul 06 19:24:07.472685 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:07.472671 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4" event={"ID":"a62f42d5-f4f7-48bf-ad8b-e87e673bac72","Type":"ContainerDied","Data":"c766e442890a3078dcfbc1564b9d94cdbbe07d89dfa5a8a51757fddf443c81ab"} Jul 06 19:24:07.472836 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:07.472709 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4" event={"ID":"a62f42d5-f4f7-48bf-ad8b-e87e673bac72","Type":"ContainerStarted","Data":"6b610b8417fb2d3c6c646837bdc21306c8554d11dbb91d7755d554e94167d746"} Jul 06 19:24:07.480524 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:07.480496 2566 generic.go:356] "Generic (PLEG): container finished" podID="ab2c8e51-541d-4f44-9d87-a64dd000f2d7" containerID="0bde38c680b921d0f6db80a668268fb938ac672cf8803757f8c1c93e935a4073" exitCode=0 Jul 06 19:24:07.480598 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:07.480544 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn" event={"ID":"ab2c8e51-541d-4f44-9d87-a64dd000f2d7","Type":"ContainerDied","Data":"0bde38c680b921d0f6db80a668268fb938ac672cf8803757f8c1c93e935a4073"} Jul 06 19:24:07.482152 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:07.482132 2566 generic.go:356] "Generic (PLEG): container finished" podID="90fd6204-edb5-4cce-aee2-c576f74414e2" containerID="ba8474df0fe7e60c5f8a269cc0eb3de8cf9b7710f3e6522bffe3bdd4f538d368" exitCode=0 Jul 06 19:24:07.482246 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:07.482201 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85" event={"ID":"90fd6204-edb5-4cce-aee2-c576f74414e2","Type":"ContainerDied","Data":"ba8474df0fe7e60c5f8a269cc0eb3de8cf9b7710f3e6522bffe3bdd4f538d368"} Jul 06 19:24:08.490855 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:08.490766 2566 generic.go:356] "Generic (PLEG): container finished" podID="90fd6204-edb5-4cce-aee2-c576f74414e2" containerID="d74c48d92b383b1add072c3ab865562535a542ac6064365ae034ce39b08256cb" exitCode=0 Jul 06 19:24:08.490855 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:08.490837 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85" event={"ID":"90fd6204-edb5-4cce-aee2-c576f74414e2","Type":"ContainerDied","Data":"d74c48d92b383b1add072c3ab865562535a542ac6064365ae034ce39b08256cb"} Jul 06 19:24:08.493356 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:08.493328 2566 generic.go:356] "Generic (PLEG): container finished" podID="da1a9d25-a8bf-4d2b-9531-c15710aa8192" containerID="a17e292e6d101f66165281bb5d06527e1abc0b8f996004097ec962590677ebc2" exitCode=0 Jul 06 19:24:08.493356 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:08.493354 2566 generic.go:356] "Generic (PLEG): container finished" podID="da1a9d25-a8bf-4d2b-9531-c15710aa8192" containerID="b88bdfc2de497e4a4edcbaca30fbd9f57d800fd52229270f4f035df3b50d474f" exitCode=0 Jul 06 19:24:08.493508 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:08.493401 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm" event={"ID":"da1a9d25-a8bf-4d2b-9531-c15710aa8192","Type":"ContainerDied","Data":"a17e292e6d101f66165281bb5d06527e1abc0b8f996004097ec962590677ebc2"} Jul 06 19:24:08.493508 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:08.493446 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm" event={"ID":"da1a9d25-a8bf-4d2b-9531-c15710aa8192","Type":"ContainerDied","Data":"b88bdfc2de497e4a4edcbaca30fbd9f57d800fd52229270f4f035df3b50d474f"} Jul 06 19:24:08.642283 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:08.642259 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn" Jul 06 19:24:08.767417 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:08.767346 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab2c8e51-541d-4f44-9d87-a64dd000f2d7-bundle\") pod \"ab2c8e51-541d-4f44-9d87-a64dd000f2d7\" (UID: \"ab2c8e51-541d-4f44-9d87-a64dd000f2d7\") " Jul 06 19:24:08.767417 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:08.767411 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btwz9\" (UniqueName: \"kubernetes.io/projected/ab2c8e51-541d-4f44-9d87-a64dd000f2d7-kube-api-access-btwz9\") pod \"ab2c8e51-541d-4f44-9d87-a64dd000f2d7\" (UID: \"ab2c8e51-541d-4f44-9d87-a64dd000f2d7\") " Jul 06 19:24:08.767609 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:08.767440 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab2c8e51-541d-4f44-9d87-a64dd000f2d7-util\") pod \"ab2c8e51-541d-4f44-9d87-a64dd000f2d7\" (UID: \"ab2c8e51-541d-4f44-9d87-a64dd000f2d7\") " Jul 06 19:24:08.767962 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:08.767937 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2c8e51-541d-4f44-9d87-a64dd000f2d7-bundle" (OuterVolumeSpecName: "bundle") pod "ab2c8e51-541d-4f44-9d87-a64dd000f2d7" (UID: "ab2c8e51-541d-4f44-9d87-a64dd000f2d7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:24:08.769488 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:08.769461 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab2c8e51-541d-4f44-9d87-a64dd000f2d7-kube-api-access-btwz9" (OuterVolumeSpecName: "kube-api-access-btwz9") pod "ab2c8e51-541d-4f44-9d87-a64dd000f2d7" (UID: "ab2c8e51-541d-4f44-9d87-a64dd000f2d7"). InnerVolumeSpecName "kube-api-access-btwz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:24:08.773252 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:08.773225 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2c8e51-541d-4f44-9d87-a64dd000f2d7-util" (OuterVolumeSpecName: "util") pod "ab2c8e51-541d-4f44-9d87-a64dd000f2d7" (UID: "ab2c8e51-541d-4f44-9d87-a64dd000f2d7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:24:08.868595 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:08.868561 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab2c8e51-541d-4f44-9d87-a64dd000f2d7-bundle\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:24:08.868595 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:08.868588 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btwz9\" (UniqueName: \"kubernetes.io/projected/ab2c8e51-541d-4f44-9d87-a64dd000f2d7-kube-api-access-btwz9\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:24:08.868595 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:08.868599 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab2c8e51-541d-4f44-9d87-a64dd000f2d7-util\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:24:09.500048 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:09.499956 2566 generic.go:356] "Generic (PLEG): container finished" podID="a62f42d5-f4f7-48bf-ad8b-e87e673bac72" containerID="02c23e49dabf83212bb0252f6577d1ac3bebeaeb27cab3c2fb3001e02f7f2b68" exitCode=0 Jul 06 19:24:09.500459 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:09.500045 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4" event={"ID":"a62f42d5-f4f7-48bf-ad8b-e87e673bac72","Type":"ContainerDied","Data":"02c23e49dabf83212bb0252f6577d1ac3bebeaeb27cab3c2fb3001e02f7f2b68"} Jul 06 19:24:09.502025 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:09.502007 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn" Jul 06 19:24:09.502137 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:09.502024 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn" event={"ID":"ab2c8e51-541d-4f44-9d87-a64dd000f2d7","Type":"ContainerDied","Data":"637dfa417e82aeaee31795ddec898264cbe01588089e0b5b54d0ae4cf217f9a0"} Jul 06 19:24:09.502137 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:09.502097 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="637dfa417e82aeaee31795ddec898264cbe01588089e0b5b54d0ae4cf217f9a0" Jul 06 19:24:09.504090 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:09.504068 2566 generic.go:356] "Generic (PLEG): container finished" podID="90fd6204-edb5-4cce-aee2-c576f74414e2" containerID="1f3c13ad3c885d30633b38e7eb663839504f091af1d7daa13829db694a08dbdb" exitCode=0 Jul 06 19:24:09.504198 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:09.504149 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85" event={"ID":"90fd6204-edb5-4cce-aee2-c576f74414e2","Type":"ContainerDied","Data":"1f3c13ad3c885d30633b38e7eb663839504f091af1d7daa13829db694a08dbdb"} Jul 06 19:24:09.636443 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:09.636420 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm" Jul 06 19:24:09.775402 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:09.775329 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfppf\" (UniqueName: \"kubernetes.io/projected/da1a9d25-a8bf-4d2b-9531-c15710aa8192-kube-api-access-zfppf\") pod \"da1a9d25-a8bf-4d2b-9531-c15710aa8192\" (UID: \"da1a9d25-a8bf-4d2b-9531-c15710aa8192\") " Jul 06 19:24:09.775402 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:09.775363 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da1a9d25-a8bf-4d2b-9531-c15710aa8192-util\") pod \"da1a9d25-a8bf-4d2b-9531-c15710aa8192\" (UID: \"da1a9d25-a8bf-4d2b-9531-c15710aa8192\") " Jul 06 19:24:09.775614 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:09.775413 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da1a9d25-a8bf-4d2b-9531-c15710aa8192-bundle\") pod \"da1a9d25-a8bf-4d2b-9531-c15710aa8192\" (UID: \"da1a9d25-a8bf-4d2b-9531-c15710aa8192\") " Jul 06 19:24:09.775908 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:09.775887 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1a9d25-a8bf-4d2b-9531-c15710aa8192-bundle" (OuterVolumeSpecName: "bundle") pod "da1a9d25-a8bf-4d2b-9531-c15710aa8192" (UID: "da1a9d25-a8bf-4d2b-9531-c15710aa8192"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:24:09.777326 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:09.777303 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1a9d25-a8bf-4d2b-9531-c15710aa8192-kube-api-access-zfppf" (OuterVolumeSpecName: "kube-api-access-zfppf") pod "da1a9d25-a8bf-4d2b-9531-c15710aa8192" (UID: "da1a9d25-a8bf-4d2b-9531-c15710aa8192"). InnerVolumeSpecName "kube-api-access-zfppf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:24:09.780587 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:09.780560 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1a9d25-a8bf-4d2b-9531-c15710aa8192-util" (OuterVolumeSpecName: "util") pod "da1a9d25-a8bf-4d2b-9531-c15710aa8192" (UID: "da1a9d25-a8bf-4d2b-9531-c15710aa8192"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:24:09.877022 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:09.876980 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da1a9d25-a8bf-4d2b-9531-c15710aa8192-bundle\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:24:09.877022 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:09.877015 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfppf\" (UniqueName: \"kubernetes.io/projected/da1a9d25-a8bf-4d2b-9531-c15710aa8192-kube-api-access-zfppf\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:24:09.877022 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:09.877029 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da1a9d25-a8bf-4d2b-9531-c15710aa8192-util\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:24:10.510417 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:10.510325 2566 generic.go:356] "Generic (PLEG): container finished" podID="a62f42d5-f4f7-48bf-ad8b-e87e673bac72" containerID="6f12b0cfdfd6bf85f2560ee590cbe14ddbde335c8638825ccc3b0c935818694b" exitCode=0 Jul 06 19:24:10.510857 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:10.510410 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4" event={"ID":"a62f42d5-f4f7-48bf-ad8b-e87e673bac72","Type":"ContainerDied","Data":"6f12b0cfdfd6bf85f2560ee590cbe14ddbde335c8638825ccc3b0c935818694b"} Jul 06 19:24:10.512223 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:10.512190 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm" event={"ID":"da1a9d25-a8bf-4d2b-9531-c15710aa8192","Type":"ContainerDied","Data":"beb39865327de43dad21bd4b6f3f1a41c7edc5f4bbd09ebb6b394e8be1414ade"} Jul 06 19:24:10.512223 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:10.512205 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm" Jul 06 19:24:10.512333 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:10.512232 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="beb39865327de43dad21bd4b6f3f1a41c7edc5f4bbd09ebb6b394e8be1414ade" Jul 06 19:24:10.644016 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:10.643994 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85" Jul 06 19:24:10.784514 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:10.784428 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90fd6204-edb5-4cce-aee2-c576f74414e2-bundle\") pod \"90fd6204-edb5-4cce-aee2-c576f74414e2\" (UID: \"90fd6204-edb5-4cce-aee2-c576f74414e2\") " Jul 06 19:24:10.784514 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:10.784513 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90fd6204-edb5-4cce-aee2-c576f74414e2-util\") pod \"90fd6204-edb5-4cce-aee2-c576f74414e2\" (UID: \"90fd6204-edb5-4cce-aee2-c576f74414e2\") " Jul 06 19:24:10.784711 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:10.784555 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztscf\" (UniqueName: \"kubernetes.io/projected/90fd6204-edb5-4cce-aee2-c576f74414e2-kube-api-access-ztscf\") pod \"90fd6204-edb5-4cce-aee2-c576f74414e2\" (UID: \"90fd6204-edb5-4cce-aee2-c576f74414e2\") " Jul 06 19:24:10.784930 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:10.784906 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90fd6204-edb5-4cce-aee2-c576f74414e2-bundle" (OuterVolumeSpecName: "bundle") pod "90fd6204-edb5-4cce-aee2-c576f74414e2" (UID: "90fd6204-edb5-4cce-aee2-c576f74414e2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:24:10.786641 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:10.786612 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90fd6204-edb5-4cce-aee2-c576f74414e2-kube-api-access-ztscf" (OuterVolumeSpecName: "kube-api-access-ztscf") pod "90fd6204-edb5-4cce-aee2-c576f74414e2" (UID: "90fd6204-edb5-4cce-aee2-c576f74414e2"). InnerVolumeSpecName "kube-api-access-ztscf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:24:10.789707 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:10.789682 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90fd6204-edb5-4cce-aee2-c576f74414e2-util" (OuterVolumeSpecName: "util") pod "90fd6204-edb5-4cce-aee2-c576f74414e2" (UID: "90fd6204-edb5-4cce-aee2-c576f74414e2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:24:10.885751 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:10.885690 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztscf\" (UniqueName: \"kubernetes.io/projected/90fd6204-edb5-4cce-aee2-c576f74414e2-kube-api-access-ztscf\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:24:10.885751 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:10.885751 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90fd6204-edb5-4cce-aee2-c576f74414e2-bundle\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:24:10.886011 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:10.885770 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90fd6204-edb5-4cce-aee2-c576f74414e2-util\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:24:11.519125 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:11.519088 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85" event={"ID":"90fd6204-edb5-4cce-aee2-c576f74414e2","Type":"ContainerDied","Data":"6b573fd16cde30684cd74a6e69ace1148b46d70a30830dffb1dd933ad2c45ac3"} Jul 06 19:24:11.519125 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:11.519122 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85" Jul 06 19:24:11.519527 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:11.519137 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b573fd16cde30684cd74a6e69ace1148b46d70a30830dffb1dd933ad2c45ac3" Jul 06 19:24:11.650576 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:11.650549 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4" Jul 06 19:24:11.794121 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:11.794021 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr55z\" (UniqueName: \"kubernetes.io/projected/a62f42d5-f4f7-48bf-ad8b-e87e673bac72-kube-api-access-xr55z\") pod \"a62f42d5-f4f7-48bf-ad8b-e87e673bac72\" (UID: \"a62f42d5-f4f7-48bf-ad8b-e87e673bac72\") " Jul 06 19:24:11.794121 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:11.794095 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a62f42d5-f4f7-48bf-ad8b-e87e673bac72-util\") pod \"a62f42d5-f4f7-48bf-ad8b-e87e673bac72\" (UID: \"a62f42d5-f4f7-48bf-ad8b-e87e673bac72\") " Jul 06 19:24:11.794389 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:11.794127 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a62f42d5-f4f7-48bf-ad8b-e87e673bac72-bundle\") pod \"a62f42d5-f4f7-48bf-ad8b-e87e673bac72\" (UID: \"a62f42d5-f4f7-48bf-ad8b-e87e673bac72\") " Jul 06 19:24:11.794889 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:11.794863 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62f42d5-f4f7-48bf-ad8b-e87e673bac72-bundle" (OuterVolumeSpecName: "bundle") pod "a62f42d5-f4f7-48bf-ad8b-e87e673bac72" (UID: "a62f42d5-f4f7-48bf-ad8b-e87e673bac72"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:24:11.796105 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:11.796079 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62f42d5-f4f7-48bf-ad8b-e87e673bac72-kube-api-access-xr55z" (OuterVolumeSpecName: "kube-api-access-xr55z") pod "a62f42d5-f4f7-48bf-ad8b-e87e673bac72" (UID: "a62f42d5-f4f7-48bf-ad8b-e87e673bac72"). InnerVolumeSpecName "kube-api-access-xr55z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:24:11.799755 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:11.799708 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62f42d5-f4f7-48bf-ad8b-e87e673bac72-util" (OuterVolumeSpecName: "util") pod "a62f42d5-f4f7-48bf-ad8b-e87e673bac72" (UID: "a62f42d5-f4f7-48bf-ad8b-e87e673bac72"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:24:11.895199 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:11.895163 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr55z\" (UniqueName: \"kubernetes.io/projected/a62f42d5-f4f7-48bf-ad8b-e87e673bac72-kube-api-access-xr55z\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:24:11.895199 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:11.895192 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a62f42d5-f4f7-48bf-ad8b-e87e673bac72-util\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:24:11.895199 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:11.895204 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a62f42d5-f4f7-48bf-ad8b-e87e673bac72-bundle\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:24:12.526017 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:12.525987 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4" Jul 06 19:24:12.526017 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:12.526007 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4" event={"ID":"a62f42d5-f4f7-48bf-ad8b-e87e673bac72","Type":"ContainerDied","Data":"6b610b8417fb2d3c6c646837bdc21306c8554d11dbb91d7755d554e94167d746"} Jul 06 19:24:12.526558 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:12.526051 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b610b8417fb2d3c6c646837bdc21306c8554d11dbb91d7755d554e94167d746" Jul 06 19:24:18.110487 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:18.110450 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:21.511249 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:21.511215 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:22.812766 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.812711 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s"] Jul 06 19:24:22.813139 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813034 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62f42d5-f4f7-48bf-ad8b-e87e673bac72" containerName="util" Jul 06 19:24:22.813139 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813045 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62f42d5-f4f7-48bf-ad8b-e87e673bac72" containerName="util" Jul 06 19:24:22.813139 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813053 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62f42d5-f4f7-48bf-ad8b-e87e673bac72" containerName="pull" Jul 06 19:24:22.813139 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813059 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62f42d5-f4f7-48bf-ad8b-e87e673bac72" containerName="pull" Jul 06 19:24:22.813139 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813070 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da1a9d25-a8bf-4d2b-9531-c15710aa8192" containerName="util" Jul 06 19:24:22.813139 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813075 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="da1a9d25-a8bf-4d2b-9531-c15710aa8192" containerName="util" Jul 06 19:24:22.813139 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813082 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90fd6204-edb5-4cce-aee2-c576f74414e2" containerName="extract" Jul 06 19:24:22.813139 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813087 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="90fd6204-edb5-4cce-aee2-c576f74414e2" containerName="extract" Jul 06 19:24:22.813139 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813095 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab2c8e51-541d-4f44-9d87-a64dd000f2d7" containerName="pull" Jul 06 19:24:22.813139 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813100 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab2c8e51-541d-4f44-9d87-a64dd000f2d7" containerName="pull" Jul 06 19:24:22.813139 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813106 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da1a9d25-a8bf-4d2b-9531-c15710aa8192" containerName="extract" Jul 06 19:24:22.813139 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813110 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="da1a9d25-a8bf-4d2b-9531-c15710aa8192" containerName="extract" Jul 06 19:24:22.813139 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813117 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab2c8e51-541d-4f44-9d87-a64dd000f2d7" containerName="extract" Jul 06 19:24:22.813139 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813122 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab2c8e51-541d-4f44-9d87-a64dd000f2d7" containerName="extract" Jul 06 19:24:22.813139 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813129 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab2c8e51-541d-4f44-9d87-a64dd000f2d7" containerName="util" Jul 06 19:24:22.813139 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813134 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab2c8e51-541d-4f44-9d87-a64dd000f2d7" containerName="util" Jul 06 19:24:22.813139 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813138 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da1a9d25-a8bf-4d2b-9531-c15710aa8192" containerName="pull" Jul 06 19:24:22.813139 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813145 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="da1a9d25-a8bf-4d2b-9531-c15710aa8192" containerName="pull" Jul 06 19:24:22.813657 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813155 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90fd6204-edb5-4cce-aee2-c576f74414e2" containerName="pull" Jul 06 19:24:22.813657 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813160 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="90fd6204-edb5-4cce-aee2-c576f74414e2" containerName="pull" Jul 06 19:24:22.813657 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813167 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90fd6204-edb5-4cce-aee2-c576f74414e2" containerName="util" Jul 06 19:24:22.813657 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813173 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="90fd6204-edb5-4cce-aee2-c576f74414e2" containerName="util" Jul 06 19:24:22.813657 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813178 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62f42d5-f4f7-48bf-ad8b-e87e673bac72" containerName="extract" Jul 06 19:24:22.813657 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813183 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62f42d5-f4f7-48bf-ad8b-e87e673bac72" containerName="extract" Jul 06 19:24:22.813657 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813235 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a62f42d5-f4f7-48bf-ad8b-e87e673bac72" containerName="extract" Jul 06 19:24:22.813657 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813243 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab2c8e51-541d-4f44-9d87-a64dd000f2d7" containerName="extract" Jul 06 19:24:22.813657 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813249 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="90fd6204-edb5-4cce-aee2-c576f74414e2" containerName="extract" Jul 06 19:24:22.813657 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.813256 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="da1a9d25-a8bf-4d2b-9531-c15710aa8192" containerName="extract" Jul 06 19:24:22.816529 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.816509 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:22.825403 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.825375 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-z2nsb\"" Jul 06 19:24:22.830446 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.830420 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s"] Jul 06 19:24:22.886206 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.886171 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ab45810-3454-4205-99ba-5c9a377992c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2tb4s\" (UID: \"0ab45810-3454-4205-99ba-5c9a377992c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:22.886378 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.886234 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-872mt\" (UniqueName: \"kubernetes.io/projected/0ab45810-3454-4205-99ba-5c9a377992c4-kube-api-access-872mt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2tb4s\" (UID: \"0ab45810-3454-4205-99ba-5c9a377992c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:22.986735 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.986684 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ab45810-3454-4205-99ba-5c9a377992c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2tb4s\" (UID: \"0ab45810-3454-4205-99ba-5c9a377992c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:22.986896 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.986796 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872mt\" (UniqueName: \"kubernetes.io/projected/0ab45810-3454-4205-99ba-5c9a377992c4-kube-api-access-872mt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2tb4s\" (UID: \"0ab45810-3454-4205-99ba-5c9a377992c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:22.987068 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.987047 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ab45810-3454-4205-99ba-5c9a377992c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2tb4s\" (UID: \"0ab45810-3454-4205-99ba-5c9a377992c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:22.995451 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:22.995416 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-872mt\" (UniqueName: \"kubernetes.io/projected/0ab45810-3454-4205-99ba-5c9a377992c4-kube-api-access-872mt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2tb4s\" (UID: \"0ab45810-3454-4205-99ba-5c9a377992c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:23.043306 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:23.043250 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:23.126525 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:23.126442 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:23.270529 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:23.270468 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s"] Jul 06 19:24:23.272975 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:24:23.272942 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab45810_3454_4205_99ba_5c9a377992c4.slice/crio-5f3645276d7defa931734462ac6b1f16818bfef9ee7f9d471f121eb8e867771f WatchSource:0}: Error finding container 5f3645276d7defa931734462ac6b1f16818bfef9ee7f9d471f121eb8e867771f: Status 404 returned error can't find the container with id 5f3645276d7defa931734462ac6b1f16818bfef9ee7f9d471f121eb8e867771f Jul 06 19:24:23.493195 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:23.493161 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:23.579874 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:23.579844 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" event={"ID":"0ab45810-3454-4205-99ba-5c9a377992c4","Type":"ContainerStarted","Data":"5f3645276d7defa931734462ac6b1f16818bfef9ee7f9d471f121eb8e867771f"} Jul 06 19:24:24.532732 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:24.532680 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:24.670539 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:24.670505 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:25.428208 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:25.428176 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-v46zz"] Jul 06 19:24:25.436250 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:25.436220 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v46zz" Jul 06 19:24:25.438854 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:25.438829 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-fvkvz\"" Jul 06 19:24:25.443146 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:25.443116 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-v46zz"] Jul 06 19:24:25.504379 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:25.504329 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzfx9\" (UniqueName: \"kubernetes.io/projected/52062462-8aad-4dc8-9d1c-bae22f5c4904-kube-api-access-vzfx9\") pod \"authorino-operator-75cc8d5cb7-v46zz\" (UID: \"52062462-8aad-4dc8-9d1c-bae22f5c4904\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-v46zz" Jul 06 19:24:25.605323 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:25.605287 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzfx9\" (UniqueName: \"kubernetes.io/projected/52062462-8aad-4dc8-9d1c-bae22f5c4904-kube-api-access-vzfx9\") pod \"authorino-operator-75cc8d5cb7-v46zz\" (UID: \"52062462-8aad-4dc8-9d1c-bae22f5c4904\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-v46zz" Jul 06 19:24:25.614257 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:25.614226 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzfx9\" (UniqueName: \"kubernetes.io/projected/52062462-8aad-4dc8-9d1c-bae22f5c4904-kube-api-access-vzfx9\") pod \"authorino-operator-75cc8d5cb7-v46zz\" (UID: \"52062462-8aad-4dc8-9d1c-bae22f5c4904\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-v46zz" Jul 06 19:24:25.749429 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:25.749326 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v46zz" Jul 06 19:24:25.899096 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:25.899068 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-v46zz"] Jul 06 19:24:25.901164 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:24:25.901119 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52062462_8aad_4dc8_9d1c_bae22f5c4904.slice/crio-47d15917c8a72d7409178e57130c755d7eb7de0603ecdb3a971af2810d42a5a2 WatchSource:0}: Error finding container 47d15917c8a72d7409178e57130c755d7eb7de0603ecdb3a971af2810d42a5a2: Status 404 returned error can't find the container with id 47d15917c8a72d7409178e57130c755d7eb7de0603ecdb3a971af2810d42a5a2 Jul 06 19:24:26.079673 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:26.079597 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:26.201207 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:26.201174 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:26.598888 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:26.598847 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v46zz" event={"ID":"52062462-8aad-4dc8-9d1c-bae22f5c4904","Type":"ContainerStarted","Data":"47d15917c8a72d7409178e57130c755d7eb7de0603ecdb3a971af2810d42a5a2"} Jul 06 19:24:27.666950 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:27.666915 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-zt5cm"] Jul 06 19:24:27.673860 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:27.673829 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zt5cm" Jul 06 19:24:27.676652 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:27.676623 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-8g8zf\"" Jul 06 19:24:27.676914 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:27.676898 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 06 19:24:27.688108 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:27.688084 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-zt5cm"] Jul 06 19:24:27.720115 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:27.720072 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhbgl\" (UniqueName: \"kubernetes.io/projected/9e96fdac-3e0c-43ba-8ed0-d94881436b66-kube-api-access-hhbgl\") pod \"dns-operator-controller-manager-864dd78c4c-zt5cm\" (UID: \"9e96fdac-3e0c-43ba-8ed0-d94881436b66\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zt5cm" Jul 06 19:24:27.821424 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:27.821382 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhbgl\" (UniqueName: \"kubernetes.io/projected/9e96fdac-3e0c-43ba-8ed0-d94881436b66-kube-api-access-hhbgl\") pod \"dns-operator-controller-manager-864dd78c4c-zt5cm\" (UID: \"9e96fdac-3e0c-43ba-8ed0-d94881436b66\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zt5cm" Jul 06 19:24:27.835094 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:27.834352 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhbgl\" (UniqueName: \"kubernetes.io/projected/9e96fdac-3e0c-43ba-8ed0-d94881436b66-kube-api-access-hhbgl\") pod \"dns-operator-controller-manager-864dd78c4c-zt5cm\" (UID: \"9e96fdac-3e0c-43ba-8ed0-d94881436b66\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zt5cm" Jul 06 19:24:27.988618 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:27.988531 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zt5cm" Jul 06 19:24:28.295139 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:28.295049 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:29.814824 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:29.814800 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk"] Jul 06 19:24:29.825896 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:29.825867 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk" Jul 06 19:24:29.829357 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:29.829338 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-qg4mh\"" Jul 06 19:24:29.834420 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:29.834380 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk"] Jul 06 19:24:29.835102 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:29.835080 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8hkl\" (UniqueName: \"kubernetes.io/projected/71816a5f-62fa-48b6-afbe-13776951465b-kube-api-access-c8hkl\") pod \"limitador-operator-controller-manager-859d7b78bd-r7jpk\" (UID: \"71816a5f-62fa-48b6-afbe-13776951465b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk" Jul 06 19:24:29.936313 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:29.936261 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8hkl\" (UniqueName: \"kubernetes.io/projected/71816a5f-62fa-48b6-afbe-13776951465b-kube-api-access-c8hkl\") pod \"limitador-operator-controller-manager-859d7b78bd-r7jpk\" (UID: \"71816a5f-62fa-48b6-afbe-13776951465b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk" Jul 06 19:24:29.949843 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:29.949677 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-zt5cm"] Jul 06 19:24:29.951975 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:29.951947 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8hkl\" (UniqueName: \"kubernetes.io/projected/71816a5f-62fa-48b6-afbe-13776951465b-kube-api-access-c8hkl\") pod \"limitador-operator-controller-manager-859d7b78bd-r7jpk\" (UID: \"71816a5f-62fa-48b6-afbe-13776951465b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk" Jul 06 19:24:29.953690 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:24:29.953663 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e96fdac_3e0c_43ba_8ed0_d94881436b66.slice/crio-e17147ef1d6cc6191cdc03325834fffeec5ed5cb8030da3cae0055d42fff0dd9 WatchSource:0}: Error finding container e17147ef1d6cc6191cdc03325834fffeec5ed5cb8030da3cae0055d42fff0dd9: Status 404 returned error can't find the container with id e17147ef1d6cc6191cdc03325834fffeec5ed5cb8030da3cae0055d42fff0dd9 Jul 06 19:24:30.134633 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:30.134540 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:30.228002 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:30.227964 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk" Jul 06 19:24:30.368901 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:30.368875 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk"] Jul 06 19:24:30.371293 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:24:30.371265 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71816a5f_62fa_48b6_afbe_13776951465b.slice/crio-6eeb4a226de3da45aed59b667f2e0824ccdd57bd8eea66d71530112d09fb7df7 WatchSource:0}: Error finding container 6eeb4a226de3da45aed59b667f2e0824ccdd57bd8eea66d71530112d09fb7df7: Status 404 returned error can't find the container with id 6eeb4a226de3da45aed59b667f2e0824ccdd57bd8eea66d71530112d09fb7df7 Jul 06 19:24:30.628846 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:30.628742 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zt5cm" event={"ID":"9e96fdac-3e0c-43ba-8ed0-d94881436b66","Type":"ContainerStarted","Data":"e17147ef1d6cc6191cdc03325834fffeec5ed5cb8030da3cae0055d42fff0dd9"} Jul 06 19:24:30.631239 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:30.631211 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk" event={"ID":"71816a5f-62fa-48b6-afbe-13776951465b","Type":"ContainerStarted","Data":"6eeb4a226de3da45aed59b667f2e0824ccdd57bd8eea66d71530112d09fb7df7"} Jul 06 19:24:30.633063 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:30.633032 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" event={"ID":"0ab45810-3454-4205-99ba-5c9a377992c4","Type":"ContainerStarted","Data":"cc149df690d1428a1f33f7f887a6c077957d8f351e4f7fd0bdb3e26738c58fcc"} Jul 06 19:24:30.633204 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:30.633174 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:30.634997 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:30.634978 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v46zz" event={"ID":"52062462-8aad-4dc8-9d1c-bae22f5c4904","Type":"ContainerStarted","Data":"7f39df807eee21ec02947232452899b5afd0782def4848a915bdc99ce1d5506c"} Jul 06 19:24:30.635118 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:30.635106 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v46zz" Jul 06 19:24:30.658919 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:30.658865 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" podStartSLOduration=2.134477935 podStartE2EDuration="8.658844681s" totalImagesPullingTime="6.524366746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:24:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:30.655673737 +0000 UTC m=+767.551563272" watchObservedRunningTime="2026-07-06 19:24:30.658844681 +0000 UTC m=+767.554734215" Jul 06 19:24:30.675460 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:30.675400 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v46zz" podStartSLOduration=1.781467466 podStartE2EDuration="5.675378384s" totalImagesPullingTime="3.893910918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:24:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:30.673643701 +0000 UTC m=+767.569533260" watchObservedRunningTime="2026-07-06 19:24:30.675378384 +0000 UTC m=+767.571267916" Jul 06 19:24:31.873023 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:31.872989 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:31.972606 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:31.972570 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:32.649260 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:32.649217 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zt5cm" event={"ID":"9e96fdac-3e0c-43ba-8ed0-d94881436b66","Type":"ContainerStarted","Data":"b494184f3c00b766ad84ae839b2d0fe7016bf2634afdbb6682a9d4ea1e79f697"} Jul 06 19:24:32.649443 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:32.649324 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zt5cm" Jul 06 19:24:32.666957 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:32.666910 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zt5cm" podStartSLOduration=3.4953742 podStartE2EDuration="5.66688844s" totalImagesPullingTime="2.17151424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:24:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:32.66398325 +0000 UTC m=+769.559872783" watchObservedRunningTime="2026-07-06 19:24:32.66688844 +0000 UTC m=+769.562777975" Jul 06 19:24:33.121243 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:33.121205 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:33.483387 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:33.483354 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:33.655498 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:33.655465 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk" event={"ID":"71816a5f-62fa-48b6-afbe-13776951465b","Type":"ContainerStarted","Data":"0e970eacc62a46fd1b8b90068d55147edf1ffd32bac9600a1adac66a117107b3"} Jul 06 19:24:33.655822 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:33.655801 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk" Jul 06 19:24:33.671349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:33.671263 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk" podStartSLOduration=2.184084754 podStartE2EDuration="4.67124537s" totalImagesPullingTime="2.487160616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:33.670637702 +0000 UTC m=+770.566527258" watchObservedRunningTime="2026-07-06 19:24:33.67124537 +0000 UTC m=+770.567134902" Jul 06 19:24:37.061785 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:37.061749 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:41.643966 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:41.643935 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v46zz" Jul 06 19:24:41.644344 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:41.644012 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:42.094498 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:42.094461 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:42.200337 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:42.200307 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:43.433870 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.433836 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s"] Jul 06 19:24:43.434319 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.434036 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" podUID="0ab45810-3454-4205-99ba-5c9a377992c4" containerName="manager" containerID="cri-o://cc149df690d1428a1f33f7f887a6c077957d8f351e4f7fd0bdb3e26738c58fcc" gracePeriod=2 Jul 06 19:24:43.437053 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.437028 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s"] Jul 06 19:24:43.437152 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:24:43.437067 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2tb4s\" is forbidden: User \"system:node:ip-10-0-133-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-201.ec2.internal' and this object" podUID="0ab45810-3454-4205-99ba-5c9a377992c4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:43.453316 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.453288 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk"] Jul 06 19:24:43.453645 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.453516 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk" podUID="71816a5f-62fa-48b6-afbe-13776951465b" containerName="manager" containerID="cri-o://0e970eacc62a46fd1b8b90068d55147edf1ffd32bac9600a1adac66a117107b3" gracePeriod=2 Jul 06 19:24:43.455740 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:24:43.455687 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2tb4s\" is forbidden: User \"system:node:ip-10-0-133-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-201.ec2.internal' and this object" podUID="0ab45810-3454-4205-99ba-5c9a377992c4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:43.456135 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.456020 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk" Jul 06 19:24:43.459651 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.459303 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25"] Jul 06 19:24:43.459819 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.459800 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ab45810-3454-4205-99ba-5c9a377992c4" containerName="manager" Jul 06 19:24:43.459912 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.459821 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ab45810-3454-4205-99ba-5c9a377992c4" containerName="manager" Jul 06 19:24:43.459965 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.459941 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ab45810-3454-4205-99ba-5c9a377992c4" containerName="manager" Jul 06 19:24:43.463574 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.463552 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25" Jul 06 19:24:43.464991 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:24:43.464960 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-r7jpk\" is forbidden: User \"system:node:ip-10-0-133-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-201.ec2.internal' and this object" podUID="71816a5f-62fa-48b6-afbe-13776951465b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk" Jul 06 19:24:43.465644 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.465620 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk"] Jul 06 19:24:43.466800 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:24:43.466779 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2tb4s\" is forbidden: User \"system:node:ip-10-0-133-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-201.ec2.internal' and this object" podUID="0ab45810-3454-4205-99ba-5c9a377992c4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:43.468664 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:24:43.468641 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-r7jpk\" is forbidden: User \"system:node:ip-10-0-133-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-201.ec2.internal' and this object" podUID="71816a5f-62fa-48b6-afbe-13776951465b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk" Jul 06 19:24:43.482896 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.482868 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25"] Jul 06 19:24:43.488186 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.488164 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccj7m"] Jul 06 19:24:43.488623 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.488607 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71816a5f-62fa-48b6-afbe-13776951465b" containerName="manager" Jul 06 19:24:43.488623 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.488624 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="71816a5f-62fa-48b6-afbe-13776951465b" containerName="manager" Jul 06 19:24:43.488787 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.488746 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="71816a5f-62fa-48b6-afbe-13776951465b" containerName="manager" Jul 06 19:24:43.492061 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.492040 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccj7m" Jul 06 19:24:43.507684 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.507658 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccj7m"] Jul 06 19:24:43.512688 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:24:43.512660 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2tb4s\" is forbidden: User \"system:node:ip-10-0-133-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-201.ec2.internal' and this object" podUID="0ab45810-3454-4205-99ba-5c9a377992c4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:43.515971 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:24:43.515941 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-r7jpk\" is forbidden: User \"system:node:ip-10-0-133-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-201.ec2.internal' and this object" podUID="71816a5f-62fa-48b6-afbe-13776951465b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk" Jul 06 19:24:43.535254 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:24:43.535217 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2tb4s\" is forbidden: User \"system:node:ip-10-0-133-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-201.ec2.internal' and this object" podUID="0ab45810-3454-4205-99ba-5c9a377992c4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:43.548665 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.548639 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnng5\" (UniqueName: \"kubernetes.io/projected/e8cb081c-5aff-41e7-8b2c-b7905d130509-kube-api-access-wnng5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-npw25\" (UID: \"e8cb081c-5aff-41e7-8b2c-b7905d130509\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25" Jul 06 19:24:43.548867 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.548679 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8cb081c-5aff-41e7-8b2c-b7905d130509-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-npw25\" (UID: \"e8cb081c-5aff-41e7-8b2c-b7905d130509\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25" Jul 06 19:24:43.548867 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.548707 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs7s5\" (UniqueName: \"kubernetes.io/projected/c3b22f65-0d6a-4bd6-82e9-32630f1411b5-kube-api-access-qs7s5\") pod \"limitador-operator-controller-manager-859d7b78bd-ccj7m\" (UID: \"c3b22f65-0d6a-4bd6-82e9-32630f1411b5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccj7m" Jul 06 19:24:43.649236 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.649204 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnng5\" (UniqueName: \"kubernetes.io/projected/e8cb081c-5aff-41e7-8b2c-b7905d130509-kube-api-access-wnng5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-npw25\" (UID: \"e8cb081c-5aff-41e7-8b2c-b7905d130509\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25" Jul 06 19:24:43.649363 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.649248 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8cb081c-5aff-41e7-8b2c-b7905d130509-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-npw25\" (UID: \"e8cb081c-5aff-41e7-8b2c-b7905d130509\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25" Jul 06 19:24:43.649363 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.649287 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs7s5\" (UniqueName: \"kubernetes.io/projected/c3b22f65-0d6a-4bd6-82e9-32630f1411b5-kube-api-access-qs7s5\") pod \"limitador-operator-controller-manager-859d7b78bd-ccj7m\" (UID: \"c3b22f65-0d6a-4bd6-82e9-32630f1411b5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccj7m" Jul 06 19:24:43.649667 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.649646 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8cb081c-5aff-41e7-8b2c-b7905d130509-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-npw25\" (UID: \"e8cb081c-5aff-41e7-8b2c-b7905d130509\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25" Jul 06 19:24:43.657734 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.657693 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zt5cm" Jul 06 19:24:43.660501 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:24:43.660469 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2tb4s\" is forbidden: User \"system:node:ip-10-0-133-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-201.ec2.internal' and this object" podUID="0ab45810-3454-4205-99ba-5c9a377992c4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:43.662899 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:24:43.662867 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-r7jpk\" is forbidden: User \"system:node:ip-10-0-133-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-201.ec2.internal' and this object" podUID="71816a5f-62fa-48b6-afbe-13776951465b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk" Jul 06 19:24:43.664120 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.664096 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs7s5\" (UniqueName: \"kubernetes.io/projected/c3b22f65-0d6a-4bd6-82e9-32630f1411b5-kube-api-access-qs7s5\") pod \"limitador-operator-controller-manager-859d7b78bd-ccj7m\" (UID: \"c3b22f65-0d6a-4bd6-82e9-32630f1411b5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccj7m" Jul 06 19:24:43.665527 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.665506 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnng5\" (UniqueName: \"kubernetes.io/projected/e8cb081c-5aff-41e7-8b2c-b7905d130509-kube-api-access-wnng5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-npw25\" (UID: \"e8cb081c-5aff-41e7-8b2c-b7905d130509\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25" Jul 06 19:24:43.702010 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.701990 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:43.704144 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.704122 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk" Jul 06 19:24:43.709040 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.709010 2566 generic.go:356] "Generic (PLEG): container finished" podID="71816a5f-62fa-48b6-afbe-13776951465b" containerID="0e970eacc62a46fd1b8b90068d55147edf1ffd32bac9600a1adac66a117107b3" exitCode=0 Jul 06 19:24:43.709040 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.709039 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk" Jul 06 19:24:43.709236 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.709108 2566 scope.go:117] "RemoveContainer" containerID="0e970eacc62a46fd1b8b90068d55147edf1ffd32bac9600a1adac66a117107b3" Jul 06 19:24:43.710383 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:24:43.710357 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2tb4s\" is forbidden: User \"system:node:ip-10-0-133-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-201.ec2.internal' and this object" podUID="0ab45810-3454-4205-99ba-5c9a377992c4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:43.710931 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.710830 2566 generic.go:356] "Generic (PLEG): container finished" podID="0ab45810-3454-4205-99ba-5c9a377992c4" containerID="cc149df690d1428a1f33f7f887a6c077957d8f351e4f7fd0bdb3e26738c58fcc" exitCode=0 Jul 06 19:24:43.710931 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.710873 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:43.714263 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:24:43.714239 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-r7jpk\" is forbidden: User \"system:node:ip-10-0-133-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-201.ec2.internal' and this object" podUID="71816a5f-62fa-48b6-afbe-13776951465b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk" Jul 06 19:24:43.716634 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:24:43.716613 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2tb4s\" is forbidden: User \"system:node:ip-10-0-133-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-201.ec2.internal' and this object" podUID="0ab45810-3454-4205-99ba-5c9a377992c4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:43.719335 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:24:43.719313 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-r7jpk\" is forbidden: User \"system:node:ip-10-0-133-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-201.ec2.internal' and this object" podUID="71816a5f-62fa-48b6-afbe-13776951465b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk" Jul 06 19:24:43.720213 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.720199 2566 scope.go:117] "RemoveContainer" containerID="0e970eacc62a46fd1b8b90068d55147edf1ffd32bac9600a1adac66a117107b3" Jul 06 19:24:43.720535 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:24:43.720518 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e970eacc62a46fd1b8b90068d55147edf1ffd32bac9600a1adac66a117107b3\": container with ID starting with 0e970eacc62a46fd1b8b90068d55147edf1ffd32bac9600a1adac66a117107b3 not found: ID does not exist" containerID="0e970eacc62a46fd1b8b90068d55147edf1ffd32bac9600a1adac66a117107b3" Jul 06 19:24:43.720582 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.720543 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e970eacc62a46fd1b8b90068d55147edf1ffd32bac9600a1adac66a117107b3"} err="failed to get container status \"0e970eacc62a46fd1b8b90068d55147edf1ffd32bac9600a1adac66a117107b3\": rpc error: code = NotFound desc = could not find container \"0e970eacc62a46fd1b8b90068d55147edf1ffd32bac9600a1adac66a117107b3\": container with ID starting with 0e970eacc62a46fd1b8b90068d55147edf1ffd32bac9600a1adac66a117107b3 not found: ID does not exist" Jul 06 19:24:43.720582 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.720558 2566 scope.go:117] "RemoveContainer" containerID="cc149df690d1428a1f33f7f887a6c077957d8f351e4f7fd0bdb3e26738c58fcc" Jul 06 19:24:43.729483 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.729461 2566 scope.go:117] "RemoveContainer" containerID="cc149df690d1428a1f33f7f887a6c077957d8f351e4f7fd0bdb3e26738c58fcc" Jul 06 19:24:43.729809 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:24:43.729790 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc149df690d1428a1f33f7f887a6c077957d8f351e4f7fd0bdb3e26738c58fcc\": container with ID starting with cc149df690d1428a1f33f7f887a6c077957d8f351e4f7fd0bdb3e26738c58fcc not found: ID does not exist" containerID="cc149df690d1428a1f33f7f887a6c077957d8f351e4f7fd0bdb3e26738c58fcc" Jul 06 19:24:43.729875 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.729817 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc149df690d1428a1f33f7f887a6c077957d8f351e4f7fd0bdb3e26738c58fcc"} err="failed to get container status \"cc149df690d1428a1f33f7f887a6c077957d8f351e4f7fd0bdb3e26738c58fcc\": rpc error: code = NotFound desc = could not find container \"cc149df690d1428a1f33f7f887a6c077957d8f351e4f7fd0bdb3e26738c58fcc\": container with ID starting with cc149df690d1428a1f33f7f887a6c077957d8f351e4f7fd0bdb3e26738c58fcc not found: ID does not exist" Jul 06 19:24:43.750307 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.750278 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ab45810-3454-4205-99ba-5c9a377992c4-extensions-socket-volume\") pod \"0ab45810-3454-4205-99ba-5c9a377992c4\" (UID: \"0ab45810-3454-4205-99ba-5c9a377992c4\") " Jul 06 19:24:43.750447 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.750326 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8hkl\" (UniqueName: \"kubernetes.io/projected/71816a5f-62fa-48b6-afbe-13776951465b-kube-api-access-c8hkl\") pod \"71816a5f-62fa-48b6-afbe-13776951465b\" (UID: \"71816a5f-62fa-48b6-afbe-13776951465b\") " Jul 06 19:24:43.750447 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.750366 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-872mt\" (UniqueName: \"kubernetes.io/projected/0ab45810-3454-4205-99ba-5c9a377992c4-kube-api-access-872mt\") pod \"0ab45810-3454-4205-99ba-5c9a377992c4\" (UID: \"0ab45810-3454-4205-99ba-5c9a377992c4\") " Jul 06 19:24:43.750833 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.750799 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab45810-3454-4205-99ba-5c9a377992c4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0ab45810-3454-4205-99ba-5c9a377992c4" (UID: "0ab45810-3454-4205-99ba-5c9a377992c4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:24:43.752417 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.752392 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71816a5f-62fa-48b6-afbe-13776951465b-kube-api-access-c8hkl" (OuterVolumeSpecName: "kube-api-access-c8hkl") pod "71816a5f-62fa-48b6-afbe-13776951465b" (UID: "71816a5f-62fa-48b6-afbe-13776951465b"). InnerVolumeSpecName "kube-api-access-c8hkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:24:43.752483 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.752427 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab45810-3454-4205-99ba-5c9a377992c4-kube-api-access-872mt" (OuterVolumeSpecName: "kube-api-access-872mt") pod "0ab45810-3454-4205-99ba-5c9a377992c4" (UID: "0ab45810-3454-4205-99ba-5c9a377992c4"). InnerVolumeSpecName "kube-api-access-872mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:24:43.762158 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:24:43.762131 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-r7jpk\" is forbidden: User \"system:node:ip-10-0-133-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-201.ec2.internal' and this object" podUID="71816a5f-62fa-48b6-afbe-13776951465b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7jpk" Jul 06 19:24:43.763332 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.763314 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ab45810-3454-4205-99ba-5c9a377992c4" path="/var/lib/kubelet/pods/0ab45810-3454-4205-99ba-5c9a377992c4/volumes" Jul 06 19:24:43.763620 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.763608 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71816a5f-62fa-48b6-afbe-13776951465b" path="/var/lib/kubelet/pods/71816a5f-62fa-48b6-afbe-13776951465b/volumes" Jul 06 19:24:43.764273 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:24:43.764251 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2tb4s\" is forbidden: User \"system:node:ip-10-0-133-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-201.ec2.internal' and this object" podUID="0ab45810-3454-4205-99ba-5c9a377992c4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2tb4s" Jul 06 19:24:43.851413 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.851378 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ab45810-3454-4205-99ba-5c9a377992c4-extensions-socket-volume\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:24:43.851413 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.851405 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8hkl\" (UniqueName: \"kubernetes.io/projected/71816a5f-62fa-48b6-afbe-13776951465b-kube-api-access-c8hkl\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:24:43.851413 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.851416 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-872mt\" (UniqueName: \"kubernetes.io/projected/0ab45810-3454-4205-99ba-5c9a377992c4-kube-api-access-872mt\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:24:43.869313 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.869288 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25" Jul 06 19:24:43.875715 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:43.875692 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccj7m" Jul 06 19:24:44.023360 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:44.023337 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25"] Jul 06 19:24:44.025118 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:24:44.025088 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8cb081c_5aff_41e7_8b2c_b7905d130509.slice/crio-531a5cac679e53397a3776e556f6b09bdd0314352b753465f02fd27d80d5a3d3 WatchSource:0}: Error finding container 531a5cac679e53397a3776e556f6b09bdd0314352b753465f02fd27d80d5a3d3: Status 404 returned error can't find the container with id 531a5cac679e53397a3776e556f6b09bdd0314352b753465f02fd27d80d5a3d3 Jul 06 19:24:44.041950 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:44.041927 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccj7m"] Jul 06 19:24:44.042999 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:24:44.042979 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b22f65_0d6a_4bd6_82e9_32630f1411b5.slice/crio-e1a1214e15652a47cbede273676ae97a2492f52fd7145e2e410116ea6c1fdb72 WatchSource:0}: Error finding container e1a1214e15652a47cbede273676ae97a2492f52fd7145e2e410116ea6c1fdb72: Status 404 returned error can't find the container with id e1a1214e15652a47cbede273676ae97a2492f52fd7145e2e410116ea6c1fdb72 Jul 06 19:24:44.616648 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:44.616611 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:44.711405 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:44.711293 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:44.717180 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:44.717150 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25" event={"ID":"e8cb081c-5aff-41e7-8b2c-b7905d130509","Type":"ContainerStarted","Data":"eb5007995fdf57c9e72828f3f5476f9ee3628f4ace9288853476f7f3717f09fc"} Jul 06 19:24:44.717344 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:44.717214 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25" event={"ID":"e8cb081c-5aff-41e7-8b2c-b7905d130509","Type":"ContainerStarted","Data":"531a5cac679e53397a3776e556f6b09bdd0314352b753465f02fd27d80d5a3d3"} Jul 06 19:24:44.717395 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:44.717349 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25" Jul 06 19:24:44.718769 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:44.718748 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccj7m" event={"ID":"c3b22f65-0d6a-4bd6-82e9-32630f1411b5","Type":"ContainerStarted","Data":"c74eef81fe226eb870544e0013fe9dd1ce477145728a7f37fbe8ffa9371d32ce"} Jul 06 19:24:44.718837 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:44.718792 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccj7m" event={"ID":"c3b22f65-0d6a-4bd6-82e9-32630f1411b5","Type":"ContainerStarted","Data":"e1a1214e15652a47cbede273676ae97a2492f52fd7145e2e410116ea6c1fdb72"} Jul 06 19:24:44.718837 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:44.718827 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccj7m" Jul 06 19:24:44.741562 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:44.741513 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25" podStartSLOduration=1.7414979590000002 podStartE2EDuration="1.741497959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:44.740195429 +0000 UTC m=+781.636084951" watchObservedRunningTime="2026-07-06 19:24:44.741497959 +0000 UTC m=+781.637387491" Jul 06 19:24:44.760909 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:44.760856 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccj7m" podStartSLOduration=1.7608325919999999 podStartE2EDuration="1.760832592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:44.760183376 +0000 UTC m=+781.656072909" watchObservedRunningTime="2026-07-06 19:24:44.760832592 +0000 UTC m=+781.656722126" Jul 06 19:24:49.270999 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:49.270942 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:49.379340 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:49.379302 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:53.069082 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:53.069047 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:53.170224 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:53.170191 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:55.728037 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:55.728006 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25" Jul 06 19:24:55.728459 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:24:55.728071 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccj7m" Jul 06 19:25:01.925877 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:01.925830 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:02.021994 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:02.021959 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:02.146498 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:02.146467 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25"] Jul 06 19:25:02.146881 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:02.146850 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25" podUID="e8cb081c-5aff-41e7-8b2c-b7905d130509" containerName="manager" containerID="cri-o://eb5007995fdf57c9e72828f3f5476f9ee3628f4ace9288853476f7f3717f09fc" gracePeriod=10 Jul 06 19:25:03.525565 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:03.525535 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25" Jul 06 19:25:03.619550 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:03.619515 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8cb081c-5aff-41e7-8b2c-b7905d130509-extensions-socket-volume\") pod \"e8cb081c-5aff-41e7-8b2c-b7905d130509\" (UID: \"e8cb081c-5aff-41e7-8b2c-b7905d130509\") " Jul 06 19:25:03.619771 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:03.619609 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnng5\" (UniqueName: \"kubernetes.io/projected/e8cb081c-5aff-41e7-8b2c-b7905d130509-kube-api-access-wnng5\") pod \"e8cb081c-5aff-41e7-8b2c-b7905d130509\" (UID: \"e8cb081c-5aff-41e7-8b2c-b7905d130509\") " Jul 06 19:25:03.619947 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:03.619926 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8cb081c-5aff-41e7-8b2c-b7905d130509-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e8cb081c-5aff-41e7-8b2c-b7905d130509" (UID: "e8cb081c-5aff-41e7-8b2c-b7905d130509"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:25:03.621779 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:03.621716 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8cb081c-5aff-41e7-8b2c-b7905d130509-kube-api-access-wnng5" (OuterVolumeSpecName: "kube-api-access-wnng5") pod "e8cb081c-5aff-41e7-8b2c-b7905d130509" (UID: "e8cb081c-5aff-41e7-8b2c-b7905d130509"). InnerVolumeSpecName "kube-api-access-wnng5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:25:03.720649 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:03.720609 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnng5\" (UniqueName: \"kubernetes.io/projected/e8cb081c-5aff-41e7-8b2c-b7905d130509-kube-api-access-wnng5\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:25:03.720649 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:03.720640 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8cb081c-5aff-41e7-8b2c-b7905d130509-extensions-socket-volume\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:25:03.818014 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:03.817978 2566 generic.go:356] "Generic (PLEG): container finished" podID="e8cb081c-5aff-41e7-8b2c-b7905d130509" containerID="eb5007995fdf57c9e72828f3f5476f9ee3628f4ace9288853476f7f3717f09fc" exitCode=0 Jul 06 19:25:03.818188 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:03.818024 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25" Jul 06 19:25:03.818188 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:03.818065 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25" event={"ID":"e8cb081c-5aff-41e7-8b2c-b7905d130509","Type":"ContainerDied","Data":"eb5007995fdf57c9e72828f3f5476f9ee3628f4ace9288853476f7f3717f09fc"} Jul 06 19:25:03.818188 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:03.818119 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25" event={"ID":"e8cb081c-5aff-41e7-8b2c-b7905d130509","Type":"ContainerDied","Data":"531a5cac679e53397a3776e556f6b09bdd0314352b753465f02fd27d80d5a3d3"} Jul 06 19:25:03.818188 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:03.818144 2566 scope.go:117] "RemoveContainer" containerID="eb5007995fdf57c9e72828f3f5476f9ee3628f4ace9288853476f7f3717f09fc" Jul 06 19:25:03.826888 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:03.826869 2566 scope.go:117] "RemoveContainer" containerID="eb5007995fdf57c9e72828f3f5476f9ee3628f4ace9288853476f7f3717f09fc" Jul 06 19:25:03.827149 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:25:03.827130 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb5007995fdf57c9e72828f3f5476f9ee3628f4ace9288853476f7f3717f09fc\": container with ID starting with eb5007995fdf57c9e72828f3f5476f9ee3628f4ace9288853476f7f3717f09fc not found: ID does not exist" containerID="eb5007995fdf57c9e72828f3f5476f9ee3628f4ace9288853476f7f3717f09fc" Jul 06 19:25:03.827212 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:03.827162 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5007995fdf57c9e72828f3f5476f9ee3628f4ace9288853476f7f3717f09fc"} err="failed to get container status \"eb5007995fdf57c9e72828f3f5476f9ee3628f4ace9288853476f7f3717f09fc\": rpc error: code = NotFound desc = could not find container \"eb5007995fdf57c9e72828f3f5476f9ee3628f4ace9288853476f7f3717f09fc\": container with ID starting with eb5007995fdf57c9e72828f3f5476f9ee3628f4ace9288853476f7f3717f09fc not found: ID does not exist" Jul 06 19:25:03.837079 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:03.837046 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25"] Jul 06 19:25:03.841500 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:03.841480 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-npw25"] Jul 06 19:25:04.365381 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:04.365347 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:04.489374 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:04.489339 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:05.763304 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:05.763224 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8cb081c-5aff-41e7-8b2c-b7905d130509" path="/var/lib/kubelet/pods/e8cb081c-5aff-41e7-8b2c-b7905d130509/volumes" Jul 06 19:25:09.048929 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:09.048894 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:09.150122 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:09.150082 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:13.262749 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:13.262699 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:13.336621 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:13.336572 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:18.668231 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.668198 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h"] Jul 06 19:25:18.668602 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.668552 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8cb081c-5aff-41e7-8b2c-b7905d130509" containerName="manager" Jul 06 19:25:18.668602 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.668563 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8cb081c-5aff-41e7-8b2c-b7905d130509" containerName="manager" Jul 06 19:25:18.668700 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.668616 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8cb081c-5aff-41e7-8b2c-b7905d130509" containerName="manager" Jul 06 19:25:18.670456 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.670440 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.673972 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.673950 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-q45lh\"" Jul 06 19:25:18.687644 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.687618 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h"] Jul 06 19:25:18.700815 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.700773 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:18.748103 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.748059 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91828ea8-792a-4816-a022-1b6bb2e8c19c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.748265 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.748147 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91828ea8-792a-4816-a022-1b6bb2e8c19c-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.748265 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.748173 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91828ea8-792a-4816-a022-1b6bb2e8c19c-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.748265 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.748206 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91828ea8-792a-4816-a022-1b6bb2e8c19c-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.748371 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.748296 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91828ea8-792a-4816-a022-1b6bb2e8c19c-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.748371 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.748343 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91828ea8-792a-4816-a022-1b6bb2e8c19c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.748371 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.748365 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91828ea8-792a-4816-a022-1b6bb2e8c19c-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.748471 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.748382 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzcxx\" (UniqueName: \"kubernetes.io/projected/91828ea8-792a-4816-a022-1b6bb2e8c19c-kube-api-access-lzcxx\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.748471 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.748407 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91828ea8-792a-4816-a022-1b6bb2e8c19c-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.849576 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.849528 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91828ea8-792a-4816-a022-1b6bb2e8c19c-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.849576 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.849582 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91828ea8-792a-4816-a022-1b6bb2e8c19c-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.849844 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.849636 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91828ea8-792a-4816-a022-1b6bb2e8c19c-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.849844 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.849665 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91828ea8-792a-4816-a022-1b6bb2e8c19c-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.849844 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.849708 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91828ea8-792a-4816-a022-1b6bb2e8c19c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.849844 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.849753 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91828ea8-792a-4816-a022-1b6bb2e8c19c-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.849844 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.849781 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzcxx\" (UniqueName: \"kubernetes.io/projected/91828ea8-792a-4816-a022-1b6bb2e8c19c-kube-api-access-lzcxx\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.849844 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.849811 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91828ea8-792a-4816-a022-1b6bb2e8c19c-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.850218 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.849848 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91828ea8-792a-4816-a022-1b6bb2e8c19c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.850218 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.850008 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91828ea8-792a-4816-a022-1b6bb2e8c19c-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.850218 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.850116 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91828ea8-792a-4816-a022-1b6bb2e8c19c-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.850352 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.850238 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91828ea8-792a-4816-a022-1b6bb2e8c19c-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.850352 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.850342 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91828ea8-792a-4816-a022-1b6bb2e8c19c-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.850487 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.850465 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91828ea8-792a-4816-a022-1b6bb2e8c19c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.852704 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.852682 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91828ea8-792a-4816-a022-1b6bb2e8c19c-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.852857 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.852841 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91828ea8-792a-4816-a022-1b6bb2e8c19c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.869741 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.869684 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91828ea8-792a-4816-a022-1b6bb2e8c19c-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.869891 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.869835 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzcxx\" (UniqueName: \"kubernetes.io/projected/91828ea8-792a-4816-a022-1b6bb2e8c19c-kube-api-access-lzcxx\") pod \"maas-default-gateway-openshift-default-56654dcf7-f9c2h\" (UID: \"91828ea8-792a-4816-a022-1b6bb2e8c19c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:18.981593 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:18.981505 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:19.142395 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:19.142371 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h"] Jul 06 19:25:19.144625 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:25:19.144593 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91828ea8_792a_4816_a022_1b6bb2e8c19c.slice/crio-b563b207b83bf29b76f4579387d5af70688c7a5335d7044caa8512b0d2d02dd4 WatchSource:0}: Error finding container b563b207b83bf29b76f4579387d5af70688c7a5335d7044caa8512b0d2d02dd4: Status 404 returned error can't find the container with id b563b207b83bf29b76f4579387d5af70688c7a5335d7044caa8512b0d2d02dd4 Jul 06 19:25:19.146826 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:19.146790 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:25:19.146910 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:19.146861 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:25:19.146910 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:19.146898 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:25:19.199035 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:19.198996 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:19.904174 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:19.904134 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" event={"ID":"91828ea8-792a-4816-a022-1b6bb2e8c19c","Type":"ContainerStarted","Data":"b85adec54ed330312b1c7c7389ca441a97864c4b5c8e1c04545323e10d3443b5"} Jul 06 19:25:19.904552 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:19.904184 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" event={"ID":"91828ea8-792a-4816-a022-1b6bb2e8c19c","Type":"ContainerStarted","Data":"b563b207b83bf29b76f4579387d5af70688c7a5335d7044caa8512b0d2d02dd4"} Jul 06 19:25:19.931315 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:19.931263 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" podStartSLOduration=1.931242922 podStartE2EDuration="1.931242922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:25:19.928673226 +0000 UTC m=+816.824562752" watchObservedRunningTime="2026-07-06 19:25:19.931242922 +0000 UTC m=+816.827132456" Jul 06 19:25:19.981617 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:19.981576 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:20.986831 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:20.986803 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:21.030661 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:21.030624 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:21.155278 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:21.155244 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:21.882949 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:21.882916 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:21.915109 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:21.915084 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:21.916059 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:21.916043 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:25:22.012595 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:22.012559 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:24.125783 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:24.125693 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:24.138560 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:24.138530 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cj99q"] Jul 06 19:25:24.141058 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:24.141041 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cj99q" Jul 06 19:25:24.145022 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:24.145005 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jfkvd\"" Jul 06 19:25:24.145194 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:24.145173 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 19:25:24.154206 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:24.154183 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cj99q"] Jul 06 19:25:24.241702 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:24.241665 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cj99q"] Jul 06 19:25:24.248353 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:24.248327 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:24.298419 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:24.298384 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhtwp\" (UniqueName: \"kubernetes.io/projected/8285a3c8-65ed-4063-91fa-81ebabd87384-kube-api-access-rhtwp\") pod \"limitador-limitador-64c94fd64f-cj99q\" (UID: \"8285a3c8-65ed-4063-91fa-81ebabd87384\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cj99q" Jul 06 19:25:24.298590 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:24.298437 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8285a3c8-65ed-4063-91fa-81ebabd87384-config-file\") pod \"limitador-limitador-64c94fd64f-cj99q\" (UID: \"8285a3c8-65ed-4063-91fa-81ebabd87384\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cj99q" Jul 06 19:25:24.329539 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:24.329506 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:24.399760 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:24.399647 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhtwp\" (UniqueName: \"kubernetes.io/projected/8285a3c8-65ed-4063-91fa-81ebabd87384-kube-api-access-rhtwp\") pod \"limitador-limitador-64c94fd64f-cj99q\" (UID: \"8285a3c8-65ed-4063-91fa-81ebabd87384\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cj99q" Jul 06 19:25:24.399760 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:24.399701 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8285a3c8-65ed-4063-91fa-81ebabd87384-config-file\") pod \"limitador-limitador-64c94fd64f-cj99q\" (UID: \"8285a3c8-65ed-4063-91fa-81ebabd87384\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cj99q" Jul 06 19:25:24.400365 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:24.400347 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8285a3c8-65ed-4063-91fa-81ebabd87384-config-file\") pod \"limitador-limitador-64c94fd64f-cj99q\" (UID: \"8285a3c8-65ed-4063-91fa-81ebabd87384\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cj99q" Jul 06 19:25:24.408960 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:24.408933 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhtwp\" (UniqueName: \"kubernetes.io/projected/8285a3c8-65ed-4063-91fa-81ebabd87384-kube-api-access-rhtwp\") pod \"limitador-limitador-64c94fd64f-cj99q\" (UID: \"8285a3c8-65ed-4063-91fa-81ebabd87384\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cj99q" Jul 06 19:25:24.429576 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:24.429542 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:24.451387 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:24.451359 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cj99q" Jul 06 19:25:24.625965 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:24.625934 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cj99q"] Jul 06 19:25:24.627621 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:25:24.627594 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8285a3c8_65ed_4063_91fa_81ebabd87384.slice/crio-ab38fb11c9e4a02d4dcb6981f90aca7023bd140992bec203e89d9b3a5ec24fb3 WatchSource:0}: Error finding container ab38fb11c9e4a02d4dcb6981f90aca7023bd140992bec203e89d9b3a5ec24fb3: Status 404 returned error can't find the container with id ab38fb11c9e4a02d4dcb6981f90aca7023bd140992bec203e89d9b3a5ec24fb3 Jul 06 19:25:24.939122 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:24.939084 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cj99q" event={"ID":"8285a3c8-65ed-4063-91fa-81ebabd87384","Type":"ContainerStarted","Data":"ab38fb11c9e4a02d4dcb6981f90aca7023bd140992bec203e89d9b3a5ec24fb3"} Jul 06 19:25:25.073924 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:25.073890 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-qhxqz"] Jul 06 19:25:25.076774 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:25.076650 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qhxqz" Jul 06 19:25:25.079655 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:25.079634 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fvqqr\"" Jul 06 19:25:25.096062 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:25.096026 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qhxqz"] Jul 06 19:25:25.126244 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:25.126201 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:25.206586 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:25.206508 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9qcr\" (UniqueName: \"kubernetes.io/projected/f3bf5c0b-8c23-48b7-a07d-4e45f0415b85-kube-api-access-w9qcr\") pod \"authorino-58cd955675-qhxqz\" (UID: \"f3bf5c0b-8c23-48b7-a07d-4e45f0415b85\") " pod="kuadrant-system/authorino-58cd955675-qhxqz" Jul 06 19:25:25.221389 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:25.221355 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:25.307283 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:25.307243 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9qcr\" (UniqueName: \"kubernetes.io/projected/f3bf5c0b-8c23-48b7-a07d-4e45f0415b85-kube-api-access-w9qcr\") pod \"authorino-58cd955675-qhxqz\" (UID: \"f3bf5c0b-8c23-48b7-a07d-4e45f0415b85\") " pod="kuadrant-system/authorino-58cd955675-qhxqz" Jul 06 19:25:25.317785 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:25.317703 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9qcr\" (UniqueName: \"kubernetes.io/projected/f3bf5c0b-8c23-48b7-a07d-4e45f0415b85-kube-api-access-w9qcr\") pod \"authorino-58cd955675-qhxqz\" (UID: \"f3bf5c0b-8c23-48b7-a07d-4e45f0415b85\") " pod="kuadrant-system/authorino-58cd955675-qhxqz" Jul 06 19:25:25.386706 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:25.386667 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qhxqz" Jul 06 19:25:25.559019 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:25.558989 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qhxqz"] Jul 06 19:25:25.561747 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:25:25.561701 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3bf5c0b_8c23_48b7_a07d_4e45f0415b85.slice/crio-991d36800c8a93a982bbc2dc9abda93b3f5a5b2f521d8a7fea2291fb1efb967f WatchSource:0}: Error finding container 991d36800c8a93a982bbc2dc9abda93b3f5a5b2f521d8a7fea2291fb1efb967f: Status 404 returned error can't find the container with id 991d36800c8a93a982bbc2dc9abda93b3f5a5b2f521d8a7fea2291fb1efb967f Jul 06 19:25:25.946079 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:25.946048 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qhxqz" event={"ID":"f3bf5c0b-8c23-48b7-a07d-4e45f0415b85","Type":"ContainerStarted","Data":"991d36800c8a93a982bbc2dc9abda93b3f5a5b2f521d8a7fea2291fb1efb967f"} Jul 06 19:25:29.096000 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:29.095966 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:29.227899 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:29.227862 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:29.973495 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:29.973460 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cj99q" event={"ID":"8285a3c8-65ed-4063-91fa-81ebabd87384","Type":"ContainerStarted","Data":"c42383a82ee1f64435f7ef5592d6573499ba6169cd2099d5bd0397b622a80f50"} Jul 06 19:25:29.973959 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:29.973582 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-cj99q" Jul 06 19:25:29.974915 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:29.974894 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qhxqz" event={"ID":"f3bf5c0b-8c23-48b7-a07d-4e45f0415b85","Type":"ContainerStarted","Data":"fd7d05f51fe5cae13b05476acf2963db76d484bc684f1248640d18029a84d583"} Jul 06 19:25:30.012196 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:30.012146 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-cj99q" podStartSLOduration=1.458028004 podStartE2EDuration="6.012128684s" totalImagesPullingTime="4.55410068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:25:30.010013817 +0000 UTC m=+826.905903350" watchObservedRunningTime="2026-07-06 19:25:30.012128684 +0000 UTC m=+826.908018215" Jul 06 19:25:30.068057 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:30.067994 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-qhxqz" podStartSLOduration=1.387455786 podStartE2EDuration="5.067974468s" totalImagesPullingTime="3.680518682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:25:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:25:30.066521145 +0000 UTC m=+826.962410677" watchObservedRunningTime="2026-07-06 19:25:30.067974468 +0000 UTC m=+826.963864003" Jul 06 19:25:31.533564 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:31.533530 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qhxqz"] Jul 06 19:25:31.985860 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:31.985814 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-qhxqz" podUID="f3bf5c0b-8c23-48b7-a07d-4e45f0415b85" containerName="authorino" containerID="cri-o://fd7d05f51fe5cae13b05476acf2963db76d484bc684f1248640d18029a84d583" gracePeriod=30 Jul 06 19:25:32.259772 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:32.259703 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qhxqz" Jul 06 19:25:32.273774 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:32.273750 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9qcr\" (UniqueName: \"kubernetes.io/projected/f3bf5c0b-8c23-48b7-a07d-4e45f0415b85-kube-api-access-w9qcr\") pod \"f3bf5c0b-8c23-48b7-a07d-4e45f0415b85\" (UID: \"f3bf5c0b-8c23-48b7-a07d-4e45f0415b85\") " Jul 06 19:25:32.275653 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:32.275631 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3bf5c0b-8c23-48b7-a07d-4e45f0415b85-kube-api-access-w9qcr" (OuterVolumeSpecName: "kube-api-access-w9qcr") pod "f3bf5c0b-8c23-48b7-a07d-4e45f0415b85" (UID: "f3bf5c0b-8c23-48b7-a07d-4e45f0415b85"). InnerVolumeSpecName "kube-api-access-w9qcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:25:32.375274 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:32.375239 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9qcr\" (UniqueName: \"kubernetes.io/projected/f3bf5c0b-8c23-48b7-a07d-4e45f0415b85-kube-api-access-w9qcr\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:25:32.991667 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:32.991628 2566 generic.go:356] "Generic (PLEG): container finished" podID="f3bf5c0b-8c23-48b7-a07d-4e45f0415b85" containerID="fd7d05f51fe5cae13b05476acf2963db76d484bc684f1248640d18029a84d583" exitCode=0 Jul 06 19:25:32.991667 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:32.991667 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qhxqz" Jul 06 19:25:32.992142 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:32.991710 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qhxqz" event={"ID":"f3bf5c0b-8c23-48b7-a07d-4e45f0415b85","Type":"ContainerDied","Data":"fd7d05f51fe5cae13b05476acf2963db76d484bc684f1248640d18029a84d583"} Jul 06 19:25:32.992142 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:32.991764 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qhxqz" event={"ID":"f3bf5c0b-8c23-48b7-a07d-4e45f0415b85","Type":"ContainerDied","Data":"991d36800c8a93a982bbc2dc9abda93b3f5a5b2f521d8a7fea2291fb1efb967f"} Jul 06 19:25:32.992142 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:32.991785 2566 scope.go:117] "RemoveContainer" containerID="fd7d05f51fe5cae13b05476acf2963db76d484bc684f1248640d18029a84d583" Jul 06 19:25:33.003123 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:33.003090 2566 scope.go:117] "RemoveContainer" containerID="fd7d05f51fe5cae13b05476acf2963db76d484bc684f1248640d18029a84d583" Jul 06 19:25:33.003625 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:25:33.003593 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd7d05f51fe5cae13b05476acf2963db76d484bc684f1248640d18029a84d583\": container with ID starting with fd7d05f51fe5cae13b05476acf2963db76d484bc684f1248640d18029a84d583 not found: ID does not exist" containerID="fd7d05f51fe5cae13b05476acf2963db76d484bc684f1248640d18029a84d583" Jul 06 19:25:33.003768 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:33.003637 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd7d05f51fe5cae13b05476acf2963db76d484bc684f1248640d18029a84d583"} err="failed to get container status \"fd7d05f51fe5cae13b05476acf2963db76d484bc684f1248640d18029a84d583\": rpc error: code = NotFound desc = could not find container \"fd7d05f51fe5cae13b05476acf2963db76d484bc684f1248640d18029a84d583\": container with ID starting with fd7d05f51fe5cae13b05476acf2963db76d484bc684f1248640d18029a84d583 not found: ID does not exist" Jul 06 19:25:33.031285 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:33.031254 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qhxqz"] Jul 06 19:25:33.034529 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:33.034506 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-qhxqz"] Jul 06 19:25:33.763425 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:33.763391 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3bf5c0b-8c23-48b7-a07d-4e45f0415b85" path="/var/lib/kubelet/pods/f3bf5c0b-8c23-48b7-a07d-4e45f0415b85/volumes" Jul 06 19:25:40.982002 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:40.981967 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-cj99q" Jul 06 19:25:42.481104 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:42.481065 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cj99q"] Jul 06 19:25:42.481574 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:42.481304 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-cj99q" podUID="8285a3c8-65ed-4063-91fa-81ebabd87384" containerName="limitador" containerID="cri-o://c42383a82ee1f64435f7ef5592d6573499ba6169cd2099d5bd0397b622a80f50" gracePeriod=30 Jul 06 19:25:43.044993 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:43.044897 2566 generic.go:356] "Generic (PLEG): container finished" podID="8285a3c8-65ed-4063-91fa-81ebabd87384" containerID="c42383a82ee1f64435f7ef5592d6573499ba6169cd2099d5bd0397b622a80f50" exitCode=0 Jul 06 19:25:43.044993 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:43.044952 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cj99q" event={"ID":"8285a3c8-65ed-4063-91fa-81ebabd87384","Type":"ContainerDied","Data":"c42383a82ee1f64435f7ef5592d6573499ba6169cd2099d5bd0397b622a80f50"} Jul 06 19:25:43.430465 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:43.430438 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cj99q" Jul 06 19:25:43.470000 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:43.469967 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhtwp\" (UniqueName: \"kubernetes.io/projected/8285a3c8-65ed-4063-91fa-81ebabd87384-kube-api-access-rhtwp\") pod \"8285a3c8-65ed-4063-91fa-81ebabd87384\" (UID: \"8285a3c8-65ed-4063-91fa-81ebabd87384\") " Jul 06 19:25:43.470187 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:43.470029 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8285a3c8-65ed-4063-91fa-81ebabd87384-config-file\") pod \"8285a3c8-65ed-4063-91fa-81ebabd87384\" (UID: \"8285a3c8-65ed-4063-91fa-81ebabd87384\") " Jul 06 19:25:43.470432 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:43.470408 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8285a3c8-65ed-4063-91fa-81ebabd87384-config-file" (OuterVolumeSpecName: "config-file") pod "8285a3c8-65ed-4063-91fa-81ebabd87384" (UID: "8285a3c8-65ed-4063-91fa-81ebabd87384"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:25:43.472274 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:43.472246 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8285a3c8-65ed-4063-91fa-81ebabd87384-kube-api-access-rhtwp" (OuterVolumeSpecName: "kube-api-access-rhtwp") pod "8285a3c8-65ed-4063-91fa-81ebabd87384" (UID: "8285a3c8-65ed-4063-91fa-81ebabd87384"). InnerVolumeSpecName "kube-api-access-rhtwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:25:43.570764 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:43.570733 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhtwp\" (UniqueName: \"kubernetes.io/projected/8285a3c8-65ed-4063-91fa-81ebabd87384-kube-api-access-rhtwp\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:25:43.570764 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:43.570760 2566 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8285a3c8-65ed-4063-91fa-81ebabd87384-config-file\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:25:44.051005 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:44.050963 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cj99q" event={"ID":"8285a3c8-65ed-4063-91fa-81ebabd87384","Type":"ContainerDied","Data":"ab38fb11c9e4a02d4dcb6981f90aca7023bd140992bec203e89d9b3a5ec24fb3"} Jul 06 19:25:44.051005 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:44.051007 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cj99q" Jul 06 19:25:44.051212 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:44.051042 2566 scope.go:117] "RemoveContainer" containerID="c42383a82ee1f64435f7ef5592d6573499ba6169cd2099d5bd0397b622a80f50" Jul 06 19:25:44.073731 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:44.073695 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cj99q"] Jul 06 19:25:44.080611 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:44.080583 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cj99q"] Jul 06 19:25:45.763160 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:45.763126 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8285a3c8-65ed-4063-91fa-81ebabd87384" path="/var/lib/kubelet/pods/8285a3c8-65ed-4063-91fa-81ebabd87384/volumes" Jul 06 19:25:47.724308 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:47.724274 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-5j9s4"] Jul 06 19:25:47.724665 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:47.724653 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8285a3c8-65ed-4063-91fa-81ebabd87384" containerName="limitador" Jul 06 19:25:47.724706 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:47.724667 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8285a3c8-65ed-4063-91fa-81ebabd87384" containerName="limitador" Jul 06 19:25:47.724706 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:47.724678 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3bf5c0b-8c23-48b7-a07d-4e45f0415b85" containerName="authorino" Jul 06 19:25:47.724706 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:47.724684 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3bf5c0b-8c23-48b7-a07d-4e45f0415b85" containerName="authorino" Jul 06 19:25:47.724884 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:47.724775 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8285a3c8-65ed-4063-91fa-81ebabd87384" containerName="limitador" Jul 06 19:25:47.724884 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:47.724791 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3bf5c0b-8c23-48b7-a07d-4e45f0415b85" containerName="authorino" Jul 06 19:25:47.729162 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:47.729143 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-5j9s4" Jul 06 19:25:47.732750 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:47.732711 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 06 19:25:47.733257 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:47.733242 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nmrgn\"" Jul 06 19:25:47.742848 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:47.742827 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-5j9s4"] Jul 06 19:25:47.776212 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:47.776173 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:25:47.807330 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:47.807291 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f53cbc16-d0a4-4020-810f-2a09d6a2755b-data\") pod \"postgres-69fb4594cf-5j9s4\" (UID: \"f53cbc16-d0a4-4020-810f-2a09d6a2755b\") " pod="opendatahub/postgres-69fb4594cf-5j9s4" Jul 06 19:25:47.807454 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:47.807368 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnh5b\" (UniqueName: \"kubernetes.io/projected/f53cbc16-d0a4-4020-810f-2a09d6a2755b-kube-api-access-jnh5b\") pod \"postgres-69fb4594cf-5j9s4\" (UID: \"f53cbc16-d0a4-4020-810f-2a09d6a2755b\") " pod="opendatahub/postgres-69fb4594cf-5j9s4" Jul 06 19:25:47.908271 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:47.908237 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f53cbc16-d0a4-4020-810f-2a09d6a2755b-data\") pod \"postgres-69fb4594cf-5j9s4\" (UID: \"f53cbc16-d0a4-4020-810f-2a09d6a2755b\") " pod="opendatahub/postgres-69fb4594cf-5j9s4" Jul 06 19:25:47.908418 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:47.908291 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnh5b\" (UniqueName: \"kubernetes.io/projected/f53cbc16-d0a4-4020-810f-2a09d6a2755b-kube-api-access-jnh5b\") pod \"postgres-69fb4594cf-5j9s4\" (UID: \"f53cbc16-d0a4-4020-810f-2a09d6a2755b\") " pod="opendatahub/postgres-69fb4594cf-5j9s4" Jul 06 19:25:47.908633 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:47.908614 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f53cbc16-d0a4-4020-810f-2a09d6a2755b-data\") pod \"postgres-69fb4594cf-5j9s4\" (UID: \"f53cbc16-d0a4-4020-810f-2a09d6a2755b\") " pod="opendatahub/postgres-69fb4594cf-5j9s4" Jul 06 19:25:47.917443 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:47.917421 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnh5b\" (UniqueName: \"kubernetes.io/projected/f53cbc16-d0a4-4020-810f-2a09d6a2755b-kube-api-access-jnh5b\") pod \"postgres-69fb4594cf-5j9s4\" (UID: \"f53cbc16-d0a4-4020-810f-2a09d6a2755b\") " pod="opendatahub/postgres-69fb4594cf-5j9s4" Jul 06 19:25:48.039382 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:48.039303 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-5j9s4" Jul 06 19:25:48.180472 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:48.180442 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-5j9s4"] Jul 06 19:25:48.183904 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:25:48.183878 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53cbc16_d0a4_4020_810f_2a09d6a2755b.slice/crio-8bc4c679bcd57d5a49cb4e1c8ecea3b0e0438a7b4e080a05b28711fdefb89f8f WatchSource:0}: Error finding container 8bc4c679bcd57d5a49cb4e1c8ecea3b0e0438a7b4e080a05b28711fdefb89f8f: Status 404 returned error can't find the container with id 8bc4c679bcd57d5a49cb4e1c8ecea3b0e0438a7b4e080a05b28711fdefb89f8f Jul 06 19:25:49.078030 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:49.077995 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-5j9s4" event={"ID":"f53cbc16-d0a4-4020-810f-2a09d6a2755b","Type":"ContainerStarted","Data":"8bc4c679bcd57d5a49cb4e1c8ecea3b0e0438a7b4e080a05b28711fdefb89f8f"} Jul 06 19:25:54.113678 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:54.113636 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-5j9s4" event={"ID":"f53cbc16-d0a4-4020-810f-2a09d6a2755b","Type":"ContainerStarted","Data":"2b3a62c4e23bbb7fd4ac11e4d6b5a5d0a461461e8a4f6c7cba00ee7e4a89dc36"} Jul 06 19:25:54.114108 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:54.113752 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-5j9s4" Jul 06 19:25:54.135133 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:25:54.135070 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-5j9s4" podStartSLOduration=2.109545949 podStartE2EDuration="7.135050796s" totalImagesPullingTime="5.025504847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:25:54.133296615 +0000 UTC m=+851.029186148" watchObservedRunningTime="2026-07-06 19:25:54.135050796 +0000 UTC m=+851.030940328" Jul 06 19:26:00.147446 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:00.147418 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-5j9s4" Jul 06 19:26:00.526352 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:00.526322 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:00.757904 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:00.757873 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-nsqns"] Jul 06 19:26:00.764203 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:00.764178 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-nsqns" Jul 06 19:26:00.767093 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:00.767064 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fvqqr\"" Jul 06 19:26:00.769848 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:00.769827 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-nsqns"] Jul 06 19:26:00.917524 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:00.917487 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4pg\" (UniqueName: \"kubernetes.io/projected/0c2f6702-443d-4611-80c6-305f8221cf66-kube-api-access-tn4pg\") pod \"authorino-5d4b67c446-nsqns\" (UID: \"0c2f6702-443d-4611-80c6-305f8221cf66\") " pod="kuadrant-system/authorino-5d4b67c446-nsqns" Jul 06 19:26:00.961198 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:00.961163 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:01.018319 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.018282 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4pg\" (UniqueName: \"kubernetes.io/projected/0c2f6702-443d-4611-80c6-305f8221cf66-kube-api-access-tn4pg\") pod \"authorino-5d4b67c446-nsqns\" (UID: \"0c2f6702-443d-4611-80c6-305f8221cf66\") " pod="kuadrant-system/authorino-5d4b67c446-nsqns" Jul 06 19:26:01.019483 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.019464 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-nsqns"] Jul 06 19:26:01.019689 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:26:01.019671 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-tn4pg], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-nsqns" podUID="0c2f6702-443d-4611-80c6-305f8221cf66" Jul 06 19:26:01.046512 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.046484 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4pg\" (UniqueName: \"kubernetes.io/projected/0c2f6702-443d-4611-80c6-305f8221cf66-kube-api-access-tn4pg\") pod \"authorino-5d4b67c446-nsqns\" (UID: \"0c2f6702-443d-4611-80c6-305f8221cf66\") " pod="kuadrant-system/authorino-5d4b67c446-nsqns" Jul 06 19:26:01.073977 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.073948 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-747b667576-gzrnn"] Jul 06 19:26:01.081947 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.081920 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-747b667576-gzrnn" Jul 06 19:26:01.121391 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.121359 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-747b667576-gzrnn"] Jul 06 19:26:01.147934 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.147909 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-nsqns" Jul 06 19:26:01.156431 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.156407 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-nsqns" Jul 06 19:26:01.219537 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.219457 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdgfl\" (UniqueName: \"kubernetes.io/projected/d7b27c04-c86d-4d11-8fa1-273c847c0670-kube-api-access-sdgfl\") pod \"authorino-747b667576-gzrnn\" (UID: \"d7b27c04-c86d-4d11-8fa1-273c847c0670\") " pod="kuadrant-system/authorino-747b667576-gzrnn" Jul 06 19:26:01.320426 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.320392 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn4pg\" (UniqueName: \"kubernetes.io/projected/0c2f6702-443d-4611-80c6-305f8221cf66-kube-api-access-tn4pg\") pod \"0c2f6702-443d-4611-80c6-305f8221cf66\" (UID: \"0c2f6702-443d-4611-80c6-305f8221cf66\") " Jul 06 19:26:01.320599 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.320582 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdgfl\" (UniqueName: \"kubernetes.io/projected/d7b27c04-c86d-4d11-8fa1-273c847c0670-kube-api-access-sdgfl\") pod \"authorino-747b667576-gzrnn\" (UID: \"d7b27c04-c86d-4d11-8fa1-273c847c0670\") " pod="kuadrant-system/authorino-747b667576-gzrnn" Jul 06 19:26:01.322549 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.322523 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2f6702-443d-4611-80c6-305f8221cf66-kube-api-access-tn4pg" (OuterVolumeSpecName: "kube-api-access-tn4pg") pod "0c2f6702-443d-4611-80c6-305f8221cf66" (UID: "0c2f6702-443d-4611-80c6-305f8221cf66"). InnerVolumeSpecName "kube-api-access-tn4pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:26:01.349104 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.349075 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdgfl\" (UniqueName: \"kubernetes.io/projected/d7b27c04-c86d-4d11-8fa1-273c847c0670-kube-api-access-sdgfl\") pod \"authorino-747b667576-gzrnn\" (UID: \"d7b27c04-c86d-4d11-8fa1-273c847c0670\") " pod="kuadrant-system/authorino-747b667576-gzrnn" Jul 06 19:26:01.391551 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.391508 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-747b667576-gzrnn" Jul 06 19:26:01.421904 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.421877 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn4pg\" (UniqueName: \"kubernetes.io/projected/0c2f6702-443d-4611-80c6-305f8221cf66-kube-api-access-tn4pg\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:26:01.540638 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.540609 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-747b667576-gzrnn"] Jul 06 19:26:01.571371 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.571349 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-747b667576-gzrnn"] Jul 06 19:26:01.573110 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:26:01.573085 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7b27c04_c86d_4d11_8fa1_273c847c0670.slice/crio-01f080a360e23dbdf407f080502a8c11bd0fb48b3c5d068a739aace8e724f0bb WatchSource:0}: Error finding container 01f080a360e23dbdf407f080502a8c11bd0fb48b3c5d068a739aace8e724f0bb: Status 404 returned error can't find the container with id 01f080a360e23dbdf407f080502a8c11bd0fb48b3c5d068a739aace8e724f0bb Jul 06 19:26:01.632783 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.632747 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77fd76664-2s45x"] Jul 06 19:26:01.664694 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.664669 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77fd76664-2s45x"] Jul 06 19:26:01.664864 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.664777 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77fd76664-2s45x" Jul 06 19:26:01.672484 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.672468 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 06 19:26:01.825250 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.825212 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k86hq\" (UniqueName: \"kubernetes.io/projected/97b0f20e-b183-43bc-98a7-244ca9fae1e7-kube-api-access-k86hq\") pod \"authorino-77fd76664-2s45x\" (UID: \"97b0f20e-b183-43bc-98a7-244ca9fae1e7\") " pod="kuadrant-system/authorino-77fd76664-2s45x" Jul 06 19:26:01.825446 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.825267 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97b0f20e-b183-43bc-98a7-244ca9fae1e7-tls-cert\") pod \"authorino-77fd76664-2s45x\" (UID: \"97b0f20e-b183-43bc-98a7-244ca9fae1e7\") " pod="kuadrant-system/authorino-77fd76664-2s45x" Jul 06 19:26:01.926572 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.926538 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97b0f20e-b183-43bc-98a7-244ca9fae1e7-tls-cert\") pod \"authorino-77fd76664-2s45x\" (UID: \"97b0f20e-b183-43bc-98a7-244ca9fae1e7\") " pod="kuadrant-system/authorino-77fd76664-2s45x" Jul 06 19:26:01.926762 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.926618 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k86hq\" (UniqueName: \"kubernetes.io/projected/97b0f20e-b183-43bc-98a7-244ca9fae1e7-kube-api-access-k86hq\") pod \"authorino-77fd76664-2s45x\" (UID: \"97b0f20e-b183-43bc-98a7-244ca9fae1e7\") " pod="kuadrant-system/authorino-77fd76664-2s45x" Jul 06 19:26:01.928949 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.928929 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97b0f20e-b183-43bc-98a7-244ca9fae1e7-tls-cert\") pod \"authorino-77fd76664-2s45x\" (UID: \"97b0f20e-b183-43bc-98a7-244ca9fae1e7\") " pod="kuadrant-system/authorino-77fd76664-2s45x" Jul 06 19:26:01.938368 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.938342 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k86hq\" (UniqueName: \"kubernetes.io/projected/97b0f20e-b183-43bc-98a7-244ca9fae1e7-kube-api-access-k86hq\") pod \"authorino-77fd76664-2s45x\" (UID: \"97b0f20e-b183-43bc-98a7-244ca9fae1e7\") " pod="kuadrant-system/authorino-77fd76664-2s45x" Jul 06 19:26:01.974504 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:01.974471 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77fd76664-2s45x" Jul 06 19:26:02.155635 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:02.155597 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-747b667576-gzrnn" event={"ID":"d7b27c04-c86d-4d11-8fa1-273c847c0670","Type":"ContainerStarted","Data":"01f080a360e23dbdf407f080502a8c11bd0fb48b3c5d068a739aace8e724f0bb"} Jul 06 19:26:02.155999 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:02.155649 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-nsqns" Jul 06 19:26:02.216505 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:02.216470 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-nsqns"] Jul 06 19:26:02.227258 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:02.227224 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-nsqns"] Jul 06 19:26:02.247424 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:02.247399 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77fd76664-2s45x"] Jul 06 19:26:02.249921 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:26:02.249893 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97b0f20e_b183_43bc_98a7_244ca9fae1e7.slice/crio-91c96001a7ed0d3f7671c699fcfe37465f563f54f4f079a21102ffd391fae96c WatchSource:0}: Error finding container 91c96001a7ed0d3f7671c699fcfe37465f563f54f4f079a21102ffd391fae96c: Status 404 returned error can't find the container with id 91c96001a7ed0d3f7671c699fcfe37465f563f54f4f079a21102ffd391fae96c Jul 06 19:26:03.162521 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:03.162492 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-747b667576-gzrnn" event={"ID":"d7b27c04-c86d-4d11-8fa1-273c847c0670","Type":"ContainerStarted","Data":"219b359a0762a133b2215b2deaba529ec7018d691c7a3693bdc9c984f88f37cf"} Jul 06 19:26:03.162962 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:03.162572 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-747b667576-gzrnn" podUID="d7b27c04-c86d-4d11-8fa1-273c847c0670" containerName="authorino" containerID="cri-o://219b359a0762a133b2215b2deaba529ec7018d691c7a3693bdc9c984f88f37cf" gracePeriod=30 Jul 06 19:26:03.164124 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:03.164100 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77fd76664-2s45x" event={"ID":"97b0f20e-b183-43bc-98a7-244ca9fae1e7","Type":"ContainerStarted","Data":"546f82ea4d394f446be68df5c4214436cf67831ea0e77dd9c2e191dcdeee81ef"} Jul 06 19:26:03.164124 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:03.164128 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77fd76664-2s45x" event={"ID":"97b0f20e-b183-43bc-98a7-244ca9fae1e7","Type":"ContainerStarted","Data":"91c96001a7ed0d3f7671c699fcfe37465f563f54f4f079a21102ffd391fae96c"} Jul 06 19:26:03.185077 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:03.185029 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-747b667576-gzrnn" podStartSLOduration=1.6760162950000002 podStartE2EDuration="2.185013123s" totalImagesPullingTime="508.996828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:03.184263052 +0000 UTC m=+860.080152584" watchObservedRunningTime="2026-07-06 19:26:03.185013123 +0000 UTC m=+860.080902656" Jul 06 19:26:03.211443 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:03.211393 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-77fd76664-2s45x" podStartSLOduration=1.6812732000000001 podStartE2EDuration="2.211368515s" totalImagesPullingTime="530.095315ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:03.210141139 +0000 UTC m=+860.106030696" watchObservedRunningTime="2026-07-06 19:26:03.211368515 +0000 UTC m=+860.107258047" Jul 06 19:26:03.482506 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:03.482475 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-747b667576-gzrnn" Jul 06 19:26:03.642622 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:03.642589 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdgfl\" (UniqueName: \"kubernetes.io/projected/d7b27c04-c86d-4d11-8fa1-273c847c0670-kube-api-access-sdgfl\") pod \"d7b27c04-c86d-4d11-8fa1-273c847c0670\" (UID: \"d7b27c04-c86d-4d11-8fa1-273c847c0670\") " Jul 06 19:26:03.644644 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:03.644605 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b27c04-c86d-4d11-8fa1-273c847c0670-kube-api-access-sdgfl" (OuterVolumeSpecName: "kube-api-access-sdgfl") pod "d7b27c04-c86d-4d11-8fa1-273c847c0670" (UID: "d7b27c04-c86d-4d11-8fa1-273c847c0670"). InnerVolumeSpecName "kube-api-access-sdgfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:26:03.743691 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:03.743612 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdgfl\" (UniqueName: \"kubernetes.io/projected/d7b27c04-c86d-4d11-8fa1-273c847c0670-kube-api-access-sdgfl\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:26:03.764695 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:03.764660 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c2f6702-443d-4611-80c6-305f8221cf66" path="/var/lib/kubelet/pods/0c2f6702-443d-4611-80c6-305f8221cf66/volumes" Jul 06 19:26:04.171227 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:04.171195 2566 generic.go:356] "Generic (PLEG): container finished" podID="d7b27c04-c86d-4d11-8fa1-273c847c0670" containerID="219b359a0762a133b2215b2deaba529ec7018d691c7a3693bdc9c984f88f37cf" exitCode=0 Jul 06 19:26:04.171668 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:04.171256 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-747b667576-gzrnn" Jul 06 19:26:04.171668 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:04.171275 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-747b667576-gzrnn" event={"ID":"d7b27c04-c86d-4d11-8fa1-273c847c0670","Type":"ContainerDied","Data":"219b359a0762a133b2215b2deaba529ec7018d691c7a3693bdc9c984f88f37cf"} Jul 06 19:26:04.171668 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:04.171314 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-747b667576-gzrnn" event={"ID":"d7b27c04-c86d-4d11-8fa1-273c847c0670","Type":"ContainerDied","Data":"01f080a360e23dbdf407f080502a8c11bd0fb48b3c5d068a739aace8e724f0bb"} Jul 06 19:26:04.171668 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:04.171330 2566 scope.go:117] "RemoveContainer" containerID="219b359a0762a133b2215b2deaba529ec7018d691c7a3693bdc9c984f88f37cf" Jul 06 19:26:04.180190 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:04.180172 2566 scope.go:117] "RemoveContainer" containerID="219b359a0762a133b2215b2deaba529ec7018d691c7a3693bdc9c984f88f37cf" Jul 06 19:26:04.180471 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:26:04.180450 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"219b359a0762a133b2215b2deaba529ec7018d691c7a3693bdc9c984f88f37cf\": container with ID starting with 219b359a0762a133b2215b2deaba529ec7018d691c7a3693bdc9c984f88f37cf not found: ID does not exist" containerID="219b359a0762a133b2215b2deaba529ec7018d691c7a3693bdc9c984f88f37cf" Jul 06 19:26:04.180537 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:04.180484 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"219b359a0762a133b2215b2deaba529ec7018d691c7a3693bdc9c984f88f37cf"} err="failed to get container status \"219b359a0762a133b2215b2deaba529ec7018d691c7a3693bdc9c984f88f37cf\": rpc error: code = NotFound desc = could not find container \"219b359a0762a133b2215b2deaba529ec7018d691c7a3693bdc9c984f88f37cf\": container with ID starting with 219b359a0762a133b2215b2deaba529ec7018d691c7a3693bdc9c984f88f37cf not found: ID does not exist" Jul 06 19:26:04.232138 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:04.232109 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-747b667576-gzrnn"] Jul 06 19:26:04.246333 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:04.246305 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-747b667576-gzrnn"] Jul 06 19:26:05.763701 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:05.763671 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7b27c04-c86d-4d11-8fa1-273c847c0670" path="/var/lib/kubelet/pods/d7b27c04-c86d-4d11-8fa1-273c847c0670/volumes" Jul 06 19:26:08.171460 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.171424 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:08.219785 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.219749 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-598b8bdcf6-r4xhq"] Jul 06 19:26:08.220254 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.220240 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7b27c04-c86d-4d11-8fa1-273c847c0670" containerName="authorino" Jul 06 19:26:08.220299 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.220257 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7b27c04-c86d-4d11-8fa1-273c847c0670" containerName="authorino" Jul 06 19:26:08.220349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.220338 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7b27c04-c86d-4d11-8fa1-273c847c0670" containerName="authorino" Jul 06 19:26:08.229827 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.229799 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-598b8bdcf6-r4xhq" Jul 06 19:26:08.238703 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.238674 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 19:26:08.239014 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.238998 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 19:26:08.239082 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.239061 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rxfvz\"" Jul 06 19:26:08.260634 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.260605 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-598b8bdcf6-r4xhq"] Jul 06 19:26:08.373565 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.373528 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:08.382653 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.382626 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwj2m\" (UniqueName: \"kubernetes.io/projected/16f3b08c-55e6-4eb9-a8b5-ed35239d4495-kube-api-access-zwj2m\") pod \"maas-controller-598b8bdcf6-r4xhq\" (UID: \"16f3b08c-55e6-4eb9-a8b5-ed35239d4495\") " pod="opendatahub/maas-controller-598b8bdcf6-r4xhq" Jul 06 19:26:08.382791 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.382698 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16f3b08c-55e6-4eb9-a8b5-ed35239d4495-webhook-cert\") pod \"maas-controller-598b8bdcf6-r4xhq\" (UID: \"16f3b08c-55e6-4eb9-a8b5-ed35239d4495\") " pod="opendatahub/maas-controller-598b8bdcf6-r4xhq" Jul 06 19:26:08.483498 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.483412 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16f3b08c-55e6-4eb9-a8b5-ed35239d4495-webhook-cert\") pod \"maas-controller-598b8bdcf6-r4xhq\" (UID: \"16f3b08c-55e6-4eb9-a8b5-ed35239d4495\") " pod="opendatahub/maas-controller-598b8bdcf6-r4xhq" Jul 06 19:26:08.483498 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.483476 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwj2m\" (UniqueName: \"kubernetes.io/projected/16f3b08c-55e6-4eb9-a8b5-ed35239d4495-kube-api-access-zwj2m\") pod \"maas-controller-598b8bdcf6-r4xhq\" (UID: \"16f3b08c-55e6-4eb9-a8b5-ed35239d4495\") " pod="opendatahub/maas-controller-598b8bdcf6-r4xhq" Jul 06 19:26:08.485865 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.485846 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16f3b08c-55e6-4eb9-a8b5-ed35239d4495-webhook-cert\") pod \"maas-controller-598b8bdcf6-r4xhq\" (UID: \"16f3b08c-55e6-4eb9-a8b5-ed35239d4495\") " pod="opendatahub/maas-controller-598b8bdcf6-r4xhq" Jul 06 19:26:08.497086 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.497056 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c475d6674-hxsxn"] Jul 06 19:26:08.499484 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.499466 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c475d6674-hxsxn" Jul 06 19:26:08.512568 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.512547 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwj2m\" (UniqueName: \"kubernetes.io/projected/16f3b08c-55e6-4eb9-a8b5-ed35239d4495-kube-api-access-zwj2m\") pod \"maas-controller-598b8bdcf6-r4xhq\" (UID: \"16f3b08c-55e6-4eb9-a8b5-ed35239d4495\") " pod="opendatahub/maas-controller-598b8bdcf6-r4xhq" Jul 06 19:26:08.543394 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.543363 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-598b8bdcf6-r4xhq" Jul 06 19:26:08.549105 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.549074 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c475d6674-hxsxn"] Jul 06 19:26:08.584110 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.584079 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9qd9\" (UniqueName: \"kubernetes.io/projected/f1ca37d7-da82-4519-88e6-7d5720e471f7-kube-api-access-p9qd9\") pod \"maas-controller-6c475d6674-hxsxn\" (UID: \"f1ca37d7-da82-4519-88e6-7d5720e471f7\") " pod="opendatahub/maas-controller-6c475d6674-hxsxn" Jul 06 19:26:08.584273 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.584181 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1ca37d7-da82-4519-88e6-7d5720e471f7-webhook-cert\") pod \"maas-controller-6c475d6674-hxsxn\" (UID: \"f1ca37d7-da82-4519-88e6-7d5720e471f7\") " pod="opendatahub/maas-controller-6c475d6674-hxsxn" Jul 06 19:26:08.592554 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.592517 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:08.685286 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.685248 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1ca37d7-da82-4519-88e6-7d5720e471f7-webhook-cert\") pod \"maas-controller-6c475d6674-hxsxn\" (UID: \"f1ca37d7-da82-4519-88e6-7d5720e471f7\") " pod="opendatahub/maas-controller-6c475d6674-hxsxn" Jul 06 19:26:08.685433 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.685317 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9qd9\" (UniqueName: \"kubernetes.io/projected/f1ca37d7-da82-4519-88e6-7d5720e471f7-kube-api-access-p9qd9\") pod \"maas-controller-6c475d6674-hxsxn\" (UID: \"f1ca37d7-da82-4519-88e6-7d5720e471f7\") " pod="opendatahub/maas-controller-6c475d6674-hxsxn" Jul 06 19:26:08.687891 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.687867 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1ca37d7-da82-4519-88e6-7d5720e471f7-webhook-cert\") pod \"maas-controller-6c475d6674-hxsxn\" (UID: \"f1ca37d7-da82-4519-88e6-7d5720e471f7\") " pod="opendatahub/maas-controller-6c475d6674-hxsxn" Jul 06 19:26:08.701545 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.701523 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9qd9\" (UniqueName: \"kubernetes.io/projected/f1ca37d7-da82-4519-88e6-7d5720e471f7-kube-api-access-p9qd9\") pod \"maas-controller-6c475d6674-hxsxn\" (UID: \"f1ca37d7-da82-4519-88e6-7d5720e471f7\") " pod="opendatahub/maas-controller-6c475d6674-hxsxn" Jul 06 19:26:08.702946 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.702922 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-598b8bdcf6-r4xhq"] Jul 06 19:26:08.703775 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:26:08.703754 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f3b08c_55e6_4eb9_a8b5_ed35239d4495.slice/crio-b087539fb04686fb31bbaec81714558a9731b9001aefeee2d5b0e2cf9dc3828e WatchSource:0}: Error finding container b087539fb04686fb31bbaec81714558a9731b9001aefeee2d5b0e2cf9dc3828e: Status 404 returned error can't find the container with id b087539fb04686fb31bbaec81714558a9731b9001aefeee2d5b0e2cf9dc3828e Jul 06 19:26:08.810142 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.810054 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c475d6674-hxsxn" Jul 06 19:26:08.952008 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:08.951981 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c475d6674-hxsxn"] Jul 06 19:26:08.954472 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:26:08.954448 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ca37d7_da82_4519_88e6_7d5720e471f7.slice/crio-459a079803d7f63df592be4305831881e6d4da3aa6ddf9d61fd4a63f3930e34c WatchSource:0}: Error finding container 459a079803d7f63df592be4305831881e6d4da3aa6ddf9d61fd4a63f3930e34c: Status 404 returned error can't find the container with id 459a079803d7f63df592be4305831881e6d4da3aa6ddf9d61fd4a63f3930e34c Jul 06 19:26:09.201257 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:09.201217 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-598b8bdcf6-r4xhq" event={"ID":"16f3b08c-55e6-4eb9-a8b5-ed35239d4495","Type":"ContainerStarted","Data":"b087539fb04686fb31bbaec81714558a9731b9001aefeee2d5b0e2cf9dc3828e"} Jul 06 19:26:09.202625 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:09.202593 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c475d6674-hxsxn" event={"ID":"f1ca37d7-da82-4519-88e6-7d5720e471f7","Type":"ContainerStarted","Data":"459a079803d7f63df592be4305831881e6d4da3aa6ddf9d61fd4a63f3930e34c"} Jul 06 19:26:11.102461 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:11.102373 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:11.222044 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:11.221982 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-598b8bdcf6-r4xhq" event={"ID":"16f3b08c-55e6-4eb9-a8b5-ed35239d4495","Type":"ContainerStarted","Data":"e17027adac2f895fda128ee4a7a76f4aed876ee0c6f272f54207c78814a37a14"} Jul 06 19:26:11.222288 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:11.222124 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-598b8bdcf6-r4xhq" Jul 06 19:26:11.223614 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:11.223588 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c475d6674-hxsxn" event={"ID":"f1ca37d7-da82-4519-88e6-7d5720e471f7","Type":"ContainerStarted","Data":"342b394b1b9879672ddb98b8606db642c24a1afd68cd0cb437da85b6ddc6d38c"} Jul 06 19:26:11.223764 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:11.223715 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c475d6674-hxsxn" Jul 06 19:26:11.256189 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:11.256146 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:11.264652 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:11.264604 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-598b8bdcf6-r4xhq" podStartSLOduration=1.209743153 podStartE2EDuration="3.264587971s" totalImagesPullingTime="2.054844818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:11.258767084 +0000 UTC m=+868.154656616" watchObservedRunningTime="2026-07-06 19:26:11.264587971 +0000 UTC m=+868.160477502" Jul 06 19:26:11.312775 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:11.312687 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6c475d6674-hxsxn" podStartSLOduration=1.497562595 podStartE2EDuration="3.312669337s" totalImagesPullingTime="1.815106742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:11.311230917 +0000 UTC m=+868.207120449" watchObservedRunningTime="2026-07-06 19:26:11.312669337 +0000 UTC m=+868.208558870" Jul 06 19:26:22.234431 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:22.234402 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c475d6674-hxsxn" Jul 06 19:26:22.234847 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:22.234746 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-598b8bdcf6-r4xhq" Jul 06 19:26:22.297772 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:22.297709 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-598b8bdcf6-r4xhq"] Jul 06 19:26:22.298284 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:22.298012 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-598b8bdcf6-r4xhq" podUID="16f3b08c-55e6-4eb9-a8b5-ed35239d4495" containerName="manager" containerID="cri-o://e17027adac2f895fda128ee4a7a76f4aed876ee0c6f272f54207c78814a37a14" gracePeriod=30 Jul 06 19:26:22.543971 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:22.543945 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-598b8bdcf6-r4xhq" Jul 06 19:26:22.602618 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:22.602581 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16f3b08c-55e6-4eb9-a8b5-ed35239d4495-webhook-cert\") pod \"16f3b08c-55e6-4eb9-a8b5-ed35239d4495\" (UID: \"16f3b08c-55e6-4eb9-a8b5-ed35239d4495\") " Jul 06 19:26:22.602803 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:22.602624 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwj2m\" (UniqueName: \"kubernetes.io/projected/16f3b08c-55e6-4eb9-a8b5-ed35239d4495-kube-api-access-zwj2m\") pod \"16f3b08c-55e6-4eb9-a8b5-ed35239d4495\" (UID: \"16f3b08c-55e6-4eb9-a8b5-ed35239d4495\") " Jul 06 19:26:22.604865 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:22.604840 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f3b08c-55e6-4eb9-a8b5-ed35239d4495-kube-api-access-zwj2m" (OuterVolumeSpecName: "kube-api-access-zwj2m") pod "16f3b08c-55e6-4eb9-a8b5-ed35239d4495" (UID: "16f3b08c-55e6-4eb9-a8b5-ed35239d4495"). InnerVolumeSpecName "kube-api-access-zwj2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:26:22.604969 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:22.604873 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16f3b08c-55e6-4eb9-a8b5-ed35239d4495-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "16f3b08c-55e6-4eb9-a8b5-ed35239d4495" (UID: "16f3b08c-55e6-4eb9-a8b5-ed35239d4495"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:26:22.704096 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:22.704048 2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16f3b08c-55e6-4eb9-a8b5-ed35239d4495-webhook-cert\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:26:22.704096 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:22.704089 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwj2m\" (UniqueName: \"kubernetes.io/projected/16f3b08c-55e6-4eb9-a8b5-ed35239d4495-kube-api-access-zwj2m\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:26:23.289320 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:23.289286 2566 generic.go:356] "Generic (PLEG): container finished" podID="16f3b08c-55e6-4eb9-a8b5-ed35239d4495" containerID="e17027adac2f895fda128ee4a7a76f4aed876ee0c6f272f54207c78814a37a14" exitCode=0 Jul 06 19:26:23.289817 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:23.289329 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-598b8bdcf6-r4xhq" Jul 06 19:26:23.289817 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:23.289376 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-598b8bdcf6-r4xhq" event={"ID":"16f3b08c-55e6-4eb9-a8b5-ed35239d4495","Type":"ContainerDied","Data":"e17027adac2f895fda128ee4a7a76f4aed876ee0c6f272f54207c78814a37a14"} Jul 06 19:26:23.289817 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:23.289436 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-598b8bdcf6-r4xhq" event={"ID":"16f3b08c-55e6-4eb9-a8b5-ed35239d4495","Type":"ContainerDied","Data":"b087539fb04686fb31bbaec81714558a9731b9001aefeee2d5b0e2cf9dc3828e"} Jul 06 19:26:23.289817 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:23.289470 2566 scope.go:117] "RemoveContainer" containerID="e17027adac2f895fda128ee4a7a76f4aed876ee0c6f272f54207c78814a37a14" Jul 06 19:26:23.298491 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:23.298475 2566 scope.go:117] "RemoveContainer" containerID="e17027adac2f895fda128ee4a7a76f4aed876ee0c6f272f54207c78814a37a14" Jul 06 19:26:23.298776 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:26:23.298758 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e17027adac2f895fda128ee4a7a76f4aed876ee0c6f272f54207c78814a37a14\": container with ID starting with e17027adac2f895fda128ee4a7a76f4aed876ee0c6f272f54207c78814a37a14 not found: ID does not exist" containerID="e17027adac2f895fda128ee4a7a76f4aed876ee0c6f272f54207c78814a37a14" Jul 06 19:26:23.298838 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:23.298785 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e17027adac2f895fda128ee4a7a76f4aed876ee0c6f272f54207c78814a37a14"} err="failed to get container status \"e17027adac2f895fda128ee4a7a76f4aed876ee0c6f272f54207c78814a37a14\": rpc error: code = NotFound desc = could not find container \"e17027adac2f895fda128ee4a7a76f4aed876ee0c6f272f54207c78814a37a14\": container with ID starting with e17027adac2f895fda128ee4a7a76f4aed876ee0c6f272f54207c78814a37a14 not found: ID does not exist" Jul 06 19:26:23.310756 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:23.310711 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-598b8bdcf6-r4xhq"] Jul 06 19:26:23.314412 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:23.314389 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-598b8bdcf6-r4xhq"] Jul 06 19:26:23.763549 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:23.763518 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16f3b08c-55e6-4eb9-a8b5-ed35239d4495" path="/var/lib/kubelet/pods/16f3b08c-55e6-4eb9-a8b5-ed35239d4495/volumes" Jul 06 19:26:25.743586 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:25.743545 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:37.219451 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:37.219373 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:41.373235 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:41.373189 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:41.816194 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:41.816161 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:42.090257 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.090165 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:42.275990 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.275950 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:42.376421 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.376323 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:42.386350 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.386326 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-z4dzj"] Jul 06 19:26:42.386690 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.386677 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16f3b08c-55e6-4eb9-a8b5-ed35239d4495" containerName="manager" Jul 06 19:26:42.386690 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.386691 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="16f3b08c-55e6-4eb9-a8b5-ed35239d4495" containerName="manager" Jul 06 19:26:42.386803 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.386768 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="16f3b08c-55e6-4eb9-a8b5-ed35239d4495" containerName="manager" Jul 06 19:26:42.391038 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.391009 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-z4dzj" Jul 06 19:26:42.393443 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.393421 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 19:26:42.393888 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.393864 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-fkm6k\"" Jul 06 19:26:42.400463 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.400440 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-z4dzj"] Jul 06 19:26:42.422114 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.422081 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-fhccx"] Jul 06 19:26:42.426769 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.426712 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-fhccx" Jul 06 19:26:42.437391 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.437361 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-fhccx"] Jul 06 19:26:42.468231 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.468198 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2qhh\" (UniqueName: \"kubernetes.io/projected/54d38cb0-b022-4644-a5d4-7e8e8d908019-kube-api-access-h2qhh\") pod \"payload-processing-58f86c7798-fhccx\" (UID: \"54d38cb0-b022-4644-a5d4-7e8e8d908019\") " pod="openshift-ingress/payload-processing-58f86c7798-fhccx" Jul 06 19:26:42.468392 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.468237 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b1cacc-17e2-43f2-874b-febb142224bb-plugins-config-volume\") pod \"payload-pre-processing-7964457696-z4dzj\" (UID: \"11b1cacc-17e2-43f2-874b-febb142224bb\") " pod="openshift-ingress/payload-pre-processing-7964457696-z4dzj" Jul 06 19:26:42.468392 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.468266 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54d38cb0-b022-4644-a5d4-7e8e8d908019-plugins-config-volume\") pod \"payload-processing-58f86c7798-fhccx\" (UID: \"54d38cb0-b022-4644-a5d4-7e8e8d908019\") " pod="openshift-ingress/payload-processing-58f86c7798-fhccx" Jul 06 19:26:42.468392 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.468364 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkzrf\" (UniqueName: \"kubernetes.io/projected/11b1cacc-17e2-43f2-874b-febb142224bb-kube-api-access-nkzrf\") pod \"payload-pre-processing-7964457696-z4dzj\" (UID: \"11b1cacc-17e2-43f2-874b-febb142224bb\") " pod="openshift-ingress/payload-pre-processing-7964457696-z4dzj" Jul 06 19:26:42.569705 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.569671 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2qhh\" (UniqueName: \"kubernetes.io/projected/54d38cb0-b022-4644-a5d4-7e8e8d908019-kube-api-access-h2qhh\") pod \"payload-processing-58f86c7798-fhccx\" (UID: \"54d38cb0-b022-4644-a5d4-7e8e8d908019\") " pod="openshift-ingress/payload-processing-58f86c7798-fhccx" Jul 06 19:26:42.569891 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.569711 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b1cacc-17e2-43f2-874b-febb142224bb-plugins-config-volume\") pod \"payload-pre-processing-7964457696-z4dzj\" (UID: \"11b1cacc-17e2-43f2-874b-febb142224bb\") " pod="openshift-ingress/payload-pre-processing-7964457696-z4dzj" Jul 06 19:26:42.569891 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.569780 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54d38cb0-b022-4644-a5d4-7e8e8d908019-plugins-config-volume\") pod \"payload-processing-58f86c7798-fhccx\" (UID: \"54d38cb0-b022-4644-a5d4-7e8e8d908019\") " pod="openshift-ingress/payload-processing-58f86c7798-fhccx" Jul 06 19:26:42.569891 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.569825 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkzrf\" (UniqueName: \"kubernetes.io/projected/11b1cacc-17e2-43f2-874b-febb142224bb-kube-api-access-nkzrf\") pod \"payload-pre-processing-7964457696-z4dzj\" (UID: \"11b1cacc-17e2-43f2-874b-febb142224bb\") " pod="openshift-ingress/payload-pre-processing-7964457696-z4dzj" Jul 06 19:26:42.570630 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.570610 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54d38cb0-b022-4644-a5d4-7e8e8d908019-plugins-config-volume\") pod \"payload-processing-58f86c7798-fhccx\" (UID: \"54d38cb0-b022-4644-a5d4-7e8e8d908019\") " pod="openshift-ingress/payload-processing-58f86c7798-fhccx" Jul 06 19:26:42.570682 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.570648 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b1cacc-17e2-43f2-874b-febb142224bb-plugins-config-volume\") pod \"payload-pre-processing-7964457696-z4dzj\" (UID: \"11b1cacc-17e2-43f2-874b-febb142224bb\") " pod="openshift-ingress/payload-pre-processing-7964457696-z4dzj" Jul 06 19:26:42.586431 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.586392 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkzrf\" (UniqueName: \"kubernetes.io/projected/11b1cacc-17e2-43f2-874b-febb142224bb-kube-api-access-nkzrf\") pod \"payload-pre-processing-7964457696-z4dzj\" (UID: \"11b1cacc-17e2-43f2-874b-febb142224bb\") " pod="openshift-ingress/payload-pre-processing-7964457696-z4dzj" Jul 06 19:26:42.586642 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.586627 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2qhh\" (UniqueName: \"kubernetes.io/projected/54d38cb0-b022-4644-a5d4-7e8e8d908019-kube-api-access-h2qhh\") pod \"payload-processing-58f86c7798-fhccx\" (UID: \"54d38cb0-b022-4644-a5d4-7e8e8d908019\") " pod="openshift-ingress/payload-processing-58f86c7798-fhccx" Jul 06 19:26:42.704451 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.704413 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-z4dzj" Jul 06 19:26:42.739159 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.739122 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-fhccx" Jul 06 19:26:42.856282 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.856116 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-z4dzj"] Jul 06 19:26:42.859890 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:26:42.859834 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b1cacc_17e2_43f2_874b_febb142224bb.slice/crio-81e6fa53d38a2cc0220d86df4f2e495d548402f1619b4fe2d435fc88fe0d9dcc WatchSource:0}: Error finding container 81e6fa53d38a2cc0220d86df4f2e495d548402f1619b4fe2d435fc88fe0d9dcc: Status 404 returned error can't find the container with id 81e6fa53d38a2cc0220d86df4f2e495d548402f1619b4fe2d435fc88fe0d9dcc Jul 06 19:26:42.897456 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:42.897421 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-fhccx"] Jul 06 19:26:42.899603 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:26:42.899580 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d38cb0_b022_4644_a5d4_7e8e8d908019.slice/crio-39562af2a57771895767255368f67465fd48d94785726373a8a2e16f79d360a8 WatchSource:0}: Error finding container 39562af2a57771895767255368f67465fd48d94785726373a8a2e16f79d360a8: Status 404 returned error can't find the container with id 39562af2a57771895767255368f67465fd48d94785726373a8a2e16f79d360a8 Jul 06 19:26:43.403134 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:43.403102 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-fhccx" event={"ID":"54d38cb0-b022-4644-a5d4-7e8e8d908019","Type":"ContainerStarted","Data":"39562af2a57771895767255368f67465fd48d94785726373a8a2e16f79d360a8"} Jul 06 19:26:43.404353 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:43.404327 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-z4dzj" event={"ID":"11b1cacc-17e2-43f2-874b-febb142224bb","Type":"ContainerStarted","Data":"81e6fa53d38a2cc0220d86df4f2e495d548402f1619b4fe2d435fc88fe0d9dcc"} Jul 06 19:26:43.617620 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:43.617592 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:26:43.621895 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:43.621866 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:26:45.415764 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:45.415731 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-fhccx" event={"ID":"54d38cb0-b022-4644-a5d4-7e8e8d908019","Type":"ContainerStarted","Data":"34d6bcb308e60be9a3d8d8038a9184cbc68557d9e1d9f0ebe2e1505c374a6e41"} Jul 06 19:26:45.416271 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:45.415804 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-fhccx" Jul 06 19:26:45.417172 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:45.417149 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-z4dzj" event={"ID":"11b1cacc-17e2-43f2-874b-febb142224bb","Type":"ContainerStarted","Data":"ed2cb5273f6405ec13c59a5f4119893a2795c372bddb005bfc2182199f4ad547"} Jul 06 19:26:45.417273 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:45.417258 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-z4dzj" Jul 06 19:26:45.455907 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:45.455850 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-z4dzj" podStartSLOduration=1.9045384250000001 podStartE2EDuration="3.455827074s" totalImagesPullingTime="1.551288649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:45.453936947 +0000 UTC m=+902.349826482" watchObservedRunningTime="2026-07-06 19:26:45.455827074 +0000 UTC m=+902.351716612" Jul 06 19:26:45.456229 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:45.456200 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-fhccx" podStartSLOduration=1.9413951219999999 podStartE2EDuration="3.456190779s" totalImagesPullingTime="1.514795657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:45.436007144 +0000 UTC m=+902.331896676" watchObservedRunningTime="2026-07-06 19:26:45.456190779 +0000 UTC m=+902.352080305" Jul 06 19:26:50.450215 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:50.450180 2566 generic.go:356] "Generic (PLEG): container finished" podID="91828ea8-792a-4816-a022-1b6bb2e8c19c" containerID="b85adec54ed330312b1c7c7389ca441a97864c4b5c8e1c04545323e10d3443b5" exitCode=0 Jul 06 19:26:50.450783 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:50.450236 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" event={"ID":"91828ea8-792a-4816-a022-1b6bb2e8c19c","Type":"ContainerDied","Data":"b85adec54ed330312b1c7c7389ca441a97864c4b5c8e1c04545323e10d3443b5"} Jul 06 19:26:50.450783 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:50.450605 2566 scope.go:117] "RemoveContainer" containerID="b85adec54ed330312b1c7c7389ca441a97864c4b5c8e1c04545323e10d3443b5" Jul 06 19:26:50.451617 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:50.451585 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:26:50.451745 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:50.451672 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:26:50.451745 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:50.451736 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:26:51.458740 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:51.458699 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" event={"ID":"91828ea8-792a-4816-a022-1b6bb2e8c19c","Type":"ContainerStarted","Data":"94c889a05c009b8f78311773dfe1674063d02db7c38ab7051f89b33aa2cc35ac"} Jul 06 19:26:51.915449 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:51.915423 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:26:51.982062 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:51.982022 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:26:51.983260 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:51.983235 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-f9c2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 06 19:26:51.983377 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:51.983287 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" podUID="91828ea8-792a-4816-a022-1b6bb2e8c19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 06 19:26:52.181158 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:52.181064 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:52.240422 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:52.240351 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:52.563949 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:52.563844 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:52.663860 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:52.663824 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:52.982071 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:52.982040 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-f9c2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 06 19:26:52.982248 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:52.982107 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" podUID="91828ea8-792a-4816-a022-1b6bb2e8c19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 06 19:26:53.981941 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:53.981901 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-f9c2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 06 19:26:53.982337 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:53.981972 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" podUID="91828ea8-792a-4816-a022-1b6bb2e8c19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 06 19:26:54.982905 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:54.982872 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-f9c2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 06 19:26:54.983344 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:54.982944 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" podUID="91828ea8-792a-4816-a022-1b6bb2e8c19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 06 19:26:55.986219 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:55.986185 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:26:55.987178 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:55.987156 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-f9c2h" Jul 06 19:26:56.424456 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:56.424429 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-fhccx" Jul 06 19:26:56.424654 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:26:56.424610 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-z4dzj" Jul 06 19:27:01.460386 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:01.460350 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:06.007175 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:06.007121 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:11.686133 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.686098 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw"] Jul 06 19:27:11.689587 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.689564 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:11.692972 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.692945 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 19:27:11.693154 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.693137 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 19:27:11.693981 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.693961 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-h6bns\"" Jul 06 19:27:11.693981 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.693973 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 06 19:27:11.716378 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.716352 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw"] Jul 06 19:27:11.765050 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.765016 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:11.832843 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.832805 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab9c5d7e-99ec-4324-be07-9d7d00898af7-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jhjvw\" (UID: \"ab9c5d7e-99ec-4324-be07-9d7d00898af7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:11.833016 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.832869 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab9c5d7e-99ec-4324-be07-9d7d00898af7-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jhjvw\" (UID: \"ab9c5d7e-99ec-4324-be07-9d7d00898af7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:11.833016 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.832950 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab9c5d7e-99ec-4324-be07-9d7d00898af7-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jhjvw\" (UID: \"ab9c5d7e-99ec-4324-be07-9d7d00898af7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:11.833016 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.832995 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6crv\" (UniqueName: \"kubernetes.io/projected/ab9c5d7e-99ec-4324-be07-9d7d00898af7-kube-api-access-j6crv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jhjvw\" (UID: \"ab9c5d7e-99ec-4324-be07-9d7d00898af7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:11.833159 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.833064 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab9c5d7e-99ec-4324-be07-9d7d00898af7-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jhjvw\" (UID: \"ab9c5d7e-99ec-4324-be07-9d7d00898af7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:11.833159 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.833094 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab9c5d7e-99ec-4324-be07-9d7d00898af7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jhjvw\" (UID: \"ab9c5d7e-99ec-4324-be07-9d7d00898af7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:11.933826 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.933792 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab9c5d7e-99ec-4324-be07-9d7d00898af7-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jhjvw\" (UID: \"ab9c5d7e-99ec-4324-be07-9d7d00898af7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:11.934029 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.933840 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab9c5d7e-99ec-4324-be07-9d7d00898af7-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jhjvw\" (UID: \"ab9c5d7e-99ec-4324-be07-9d7d00898af7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:11.934029 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.933878 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6crv\" (UniqueName: \"kubernetes.io/projected/ab9c5d7e-99ec-4324-be07-9d7d00898af7-kube-api-access-j6crv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jhjvw\" (UID: \"ab9c5d7e-99ec-4324-be07-9d7d00898af7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:11.934029 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.933922 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab9c5d7e-99ec-4324-be07-9d7d00898af7-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jhjvw\" (UID: \"ab9c5d7e-99ec-4324-be07-9d7d00898af7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:11.934029 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.933951 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab9c5d7e-99ec-4324-be07-9d7d00898af7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jhjvw\" (UID: \"ab9c5d7e-99ec-4324-be07-9d7d00898af7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:11.934029 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.933994 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab9c5d7e-99ec-4324-be07-9d7d00898af7-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jhjvw\" (UID: \"ab9c5d7e-99ec-4324-be07-9d7d00898af7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:11.934392 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.934359 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab9c5d7e-99ec-4324-be07-9d7d00898af7-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jhjvw\" (UID: \"ab9c5d7e-99ec-4324-be07-9d7d00898af7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:11.934467 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.934414 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab9c5d7e-99ec-4324-be07-9d7d00898af7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jhjvw\" (UID: \"ab9c5d7e-99ec-4324-be07-9d7d00898af7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:11.934509 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.934470 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab9c5d7e-99ec-4324-be07-9d7d00898af7-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jhjvw\" (UID: \"ab9c5d7e-99ec-4324-be07-9d7d00898af7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:11.936433 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.936369 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab9c5d7e-99ec-4324-be07-9d7d00898af7-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jhjvw\" (UID: \"ab9c5d7e-99ec-4324-be07-9d7d00898af7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:11.936526 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.936515 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab9c5d7e-99ec-4324-be07-9d7d00898af7-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jhjvw\" (UID: \"ab9c5d7e-99ec-4324-be07-9d7d00898af7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:11.941665 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:11.941641 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6crv\" (UniqueName: \"kubernetes.io/projected/ab9c5d7e-99ec-4324-be07-9d7d00898af7-kube-api-access-j6crv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jhjvw\" (UID: \"ab9c5d7e-99ec-4324-be07-9d7d00898af7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:12.001027 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:12.000986 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:12.163948 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:12.163923 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw"] Jul 06 19:27:12.166010 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:27:12.165987 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9c5d7e_99ec_4324_be07_9d7d00898af7.slice/crio-29c3de3599f24a1ed45f9594bb9924647969a108ecf92e7f5151c59338e85844 WatchSource:0}: Error finding container 29c3de3599f24a1ed45f9594bb9924647969a108ecf92e7f5151c59338e85844: Status 404 returned error can't find the container with id 29c3de3599f24a1ed45f9594bb9924647969a108ecf92e7f5151c59338e85844 Jul 06 19:27:12.578828 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:12.578763 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" event={"ID":"ab9c5d7e-99ec-4324-be07-9d7d00898af7","Type":"ContainerStarted","Data":"29c3de3599f24a1ed45f9594bb9924647969a108ecf92e7f5151c59338e85844"} Jul 06 19:27:13.585677 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:13.585575 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" event={"ID":"ab9c5d7e-99ec-4324-be07-9d7d00898af7","Type":"ContainerStarted","Data":"71d06e0187faaee7e5349e4a27d262e0de7b372412bf4be4b287dd07b781ccda"} Jul 06 19:27:13.605018 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:13.604968 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" podStartSLOduration=1.450663909 podStartE2EDuration="2.604948887s" totalImagesPullingTime="1.154284978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:13.604129783 +0000 UTC m=+930.500019340" watchObservedRunningTime="2026-07-06 19:27:13.604948887 +0000 UTC m=+930.500838419" Jul 06 19:27:22.002104 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:22.002052 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:22.002535 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:22.002142 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:22.014572 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:22.014544 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:22.644022 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:22.643994 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jhjvw" Jul 06 19:27:24.275278 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:24.275247 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:46.238081 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.238043 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp"] Jul 06 19:27:46.241571 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.241549 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:46.244675 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.244653 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 06 19:27:46.279521 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.279494 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp"] Jul 06 19:27:46.319446 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.319416 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp\" (UID: \"b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:46.319607 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.319452 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt7jr\" (UniqueName: \"kubernetes.io/projected/b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2-kube-api-access-nt7jr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp\" (UID: \"b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:46.319607 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.319494 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp\" (UID: \"b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:46.319607 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.319518 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp\" (UID: \"b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:46.319607 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.319538 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp\" (UID: \"b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:46.319607 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.319563 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp\" (UID: \"b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:46.352477 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.352445 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:46.420337 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.420302 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp\" (UID: \"b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:46.420337 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.420337 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp\" (UID: \"b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:46.420550 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.420471 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp\" (UID: \"b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:46.420550 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.420528 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp\" (UID: \"b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:46.420652 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.420615 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp\" (UID: \"b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:46.420652 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.420637 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt7jr\" (UniqueName: \"kubernetes.io/projected/b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2-kube-api-access-nt7jr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp\" (UID: \"b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:46.420831 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.420812 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp\" (UID: \"b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:46.421030 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.420999 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp\" (UID: \"b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:46.421117 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.421000 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp\" (UID: \"b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:46.422779 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.422751 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp\" (UID: \"b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:46.423060 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.423031 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp\" (UID: \"b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:46.438052 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.438020 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt7jr\" (UniqueName: \"kubernetes.io/projected/b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2-kube-api-access-nt7jr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp\" (UID: \"b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:46.551501 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.551417 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:46.698308 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.698272 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp"] Jul 06 19:27:46.701386 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:27:46.701358 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d81d6c_9c20_4284_86c1_7a9a26c1a6d2.slice/crio-358061482f0eef7a8ab69b50a2ebddb2b0733a62a19cfd3a01a514292e4a354f WatchSource:0}: Error finding container 358061482f0eef7a8ab69b50a2ebddb2b0733a62a19cfd3a01a514292e4a354f: Status 404 returned error can't find the container with id 358061482f0eef7a8ab69b50a2ebddb2b0733a62a19cfd3a01a514292e4a354f Jul 06 19:27:46.702829 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.702813 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:27:46.770190 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:46.770155 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" event={"ID":"b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2","Type":"ContainerStarted","Data":"358061482f0eef7a8ab69b50a2ebddb2b0733a62a19cfd3a01a514292e4a354f"} Jul 06 19:27:47.776786 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:47.776753 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" event={"ID":"b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2","Type":"ContainerStarted","Data":"c57da374c149513d9d4f248fa14c19bd51afba3c9bd23f7da2094e05a6525e4c"} Jul 06 19:27:47.813558 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:47.813500 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" podStartSLOduration=1.6475315799999999 podStartE2EDuration="1.813484865s" totalImagesPullingTime="165.953285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:47.812878729 +0000 UTC m=+964.708768261" watchObservedRunningTime="2026-07-06 19:27:47.813484865 +0000 UTC m=+964.709374397" Jul 06 19:27:55.795909 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:55.795873 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:56.551994 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:56.551954 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:56.552198 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:56.552050 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:56.564651 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:56.564628 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:56.835974 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:56.835880 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp" Jul 06 19:27:58.063619 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:27:58.063586 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:07.408128 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:28:07.408048 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:23.127745 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:23.127677 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58b889c56f-smm9k"] Jul 06 19:29:23.131486 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:23.131463 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58b889c56f-smm9k" Jul 06 19:29:23.143396 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:23.143367 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58b889c56f-smm9k"] Jul 06 19:29:23.185320 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:23.185285 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a0c1539f-c35c-4a6d-8b6c-288e7b0a9ecb-tls-cert\") pod \"authorino-58b889c56f-smm9k\" (UID: \"a0c1539f-c35c-4a6d-8b6c-288e7b0a9ecb\") " pod="kuadrant-system/authorino-58b889c56f-smm9k" Jul 06 19:29:23.185491 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:23.185342 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrskt\" (UniqueName: \"kubernetes.io/projected/a0c1539f-c35c-4a6d-8b6c-288e7b0a9ecb-kube-api-access-hrskt\") pod \"authorino-58b889c56f-smm9k\" (UID: \"a0c1539f-c35c-4a6d-8b6c-288e7b0a9ecb\") " pod="kuadrant-system/authorino-58b889c56f-smm9k" Jul 06 19:29:23.285923 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:23.285891 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a0c1539f-c35c-4a6d-8b6c-288e7b0a9ecb-tls-cert\") pod \"authorino-58b889c56f-smm9k\" (UID: \"a0c1539f-c35c-4a6d-8b6c-288e7b0a9ecb\") " pod="kuadrant-system/authorino-58b889c56f-smm9k" Jul 06 19:29:23.286094 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:23.285943 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrskt\" (UniqueName: \"kubernetes.io/projected/a0c1539f-c35c-4a6d-8b6c-288e7b0a9ecb-kube-api-access-hrskt\") pod \"authorino-58b889c56f-smm9k\" (UID: \"a0c1539f-c35c-4a6d-8b6c-288e7b0a9ecb\") " pod="kuadrant-system/authorino-58b889c56f-smm9k" Jul 06 19:29:23.288294 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:23.288270 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a0c1539f-c35c-4a6d-8b6c-288e7b0a9ecb-tls-cert\") pod \"authorino-58b889c56f-smm9k\" (UID: \"a0c1539f-c35c-4a6d-8b6c-288e7b0a9ecb\") " pod="kuadrant-system/authorino-58b889c56f-smm9k" Jul 06 19:29:23.320783 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:23.320716 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrskt\" (UniqueName: \"kubernetes.io/projected/a0c1539f-c35c-4a6d-8b6c-288e7b0a9ecb-kube-api-access-hrskt\") pod \"authorino-58b889c56f-smm9k\" (UID: \"a0c1539f-c35c-4a6d-8b6c-288e7b0a9ecb\") " pod="kuadrant-system/authorino-58b889c56f-smm9k" Jul 06 19:29:23.441098 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:23.441063 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58b889c56f-smm9k" Jul 06 19:29:23.570912 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:23.570795 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:23.618199 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:29:23.618171 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0c1539f_c35c_4a6d_8b6c_288e7b0a9ecb.slice/crio-07a0d4247f798d768238420f2db22e5273932607b2cff318a1f87f4e8fd2aa71 WatchSource:0}: Error finding container 07a0d4247f798d768238420f2db22e5273932607b2cff318a1f87f4e8fd2aa71: Status 404 returned error can't find the container with id 07a0d4247f798d768238420f2db22e5273932607b2cff318a1f87f4e8fd2aa71 Jul 06 19:29:23.620346 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:23.620322 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58b889c56f-smm9k"] Jul 06 19:29:23.729642 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:23.729526 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:24.300778 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:24.300673 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58b889c56f-smm9k" event={"ID":"a0c1539f-c35c-4a6d-8b6c-288e7b0a9ecb","Type":"ContainerStarted","Data":"a1b6b4fb6b7808042b419b84e68bf62dae74d4f2ee323a5ecbfe312e0297423e"} Jul 06 19:29:24.300778 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:24.300743 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58b889c56f-smm9k" event={"ID":"a0c1539f-c35c-4a6d-8b6c-288e7b0a9ecb","Type":"ContainerStarted","Data":"07a0d4247f798d768238420f2db22e5273932607b2cff318a1f87f4e8fd2aa71"} Jul 06 19:29:24.323810 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:24.323757 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58b889c56f-smm9k" podStartSLOduration=0.919761476 podStartE2EDuration="1.323741446s" totalImagesPullingTime="403.97997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:29:24.321348394 +0000 UTC m=+1061.217237927" watchObservedRunningTime="2026-07-06 19:29:24.323741446 +0000 UTC m=+1061.219630975" Jul 06 19:29:24.370008 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:24.369973 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-77fd76664-2s45x"] Jul 06 19:29:24.370176 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:24.370148 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-77fd76664-2s45x" podUID="97b0f20e-b183-43bc-98a7-244ca9fae1e7" containerName="authorino" containerID="cri-o://546f82ea4d394f446be68df5c4214436cf67831ea0e77dd9c2e191dcdeee81ef" gracePeriod=30 Jul 06 19:29:24.640577 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:24.640552 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77fd76664-2s45x" Jul 06 19:29:24.698006 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:24.697964 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k86hq\" (UniqueName: \"kubernetes.io/projected/97b0f20e-b183-43bc-98a7-244ca9fae1e7-kube-api-access-k86hq\") pod \"97b0f20e-b183-43bc-98a7-244ca9fae1e7\" (UID: \"97b0f20e-b183-43bc-98a7-244ca9fae1e7\") " Jul 06 19:29:24.698176 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:24.698013 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97b0f20e-b183-43bc-98a7-244ca9fae1e7-tls-cert\") pod \"97b0f20e-b183-43bc-98a7-244ca9fae1e7\" (UID: \"97b0f20e-b183-43bc-98a7-244ca9fae1e7\") " Jul 06 19:29:24.700069 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:24.700034 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b0f20e-b183-43bc-98a7-244ca9fae1e7-kube-api-access-k86hq" (OuterVolumeSpecName: "kube-api-access-k86hq") pod "97b0f20e-b183-43bc-98a7-244ca9fae1e7" (UID: "97b0f20e-b183-43bc-98a7-244ca9fae1e7"). InnerVolumeSpecName "kube-api-access-k86hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:29:24.708753 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:24.708701 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97b0f20e-b183-43bc-98a7-244ca9fae1e7-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "97b0f20e-b183-43bc-98a7-244ca9fae1e7" (UID: "97b0f20e-b183-43bc-98a7-244ca9fae1e7"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:29:24.799373 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:24.799327 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k86hq\" (UniqueName: \"kubernetes.io/projected/97b0f20e-b183-43bc-98a7-244ca9fae1e7-kube-api-access-k86hq\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:29:24.799373 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:24.799360 2566 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97b0f20e-b183-43bc-98a7-244ca9fae1e7-tls-cert\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:29:24.899155 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:24.899054 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:25.082513 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:25.082479 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:25.308425 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:25.308395 2566 generic.go:356] "Generic (PLEG): container finished" podID="97b0f20e-b183-43bc-98a7-244ca9fae1e7" containerID="546f82ea4d394f446be68df5c4214436cf67831ea0e77dd9c2e191dcdeee81ef" exitCode=0 Jul 06 19:29:25.308925 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:25.308454 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77fd76664-2s45x" Jul 06 19:29:25.308925 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:25.308494 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77fd76664-2s45x" event={"ID":"97b0f20e-b183-43bc-98a7-244ca9fae1e7","Type":"ContainerDied","Data":"546f82ea4d394f446be68df5c4214436cf67831ea0e77dd9c2e191dcdeee81ef"} Jul 06 19:29:25.308925 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:25.308544 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77fd76664-2s45x" event={"ID":"97b0f20e-b183-43bc-98a7-244ca9fae1e7","Type":"ContainerDied","Data":"91c96001a7ed0d3f7671c699fcfe37465f563f54f4f079a21102ffd391fae96c"} Jul 06 19:29:25.308925 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:25.308569 2566 scope.go:117] "RemoveContainer" containerID="546f82ea4d394f446be68df5c4214436cf67831ea0e77dd9c2e191dcdeee81ef" Jul 06 19:29:25.320151 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:25.320129 2566 scope.go:117] "RemoveContainer" containerID="546f82ea4d394f446be68df5c4214436cf67831ea0e77dd9c2e191dcdeee81ef" Jul 06 19:29:25.320474 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:29:25.320425 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546f82ea4d394f446be68df5c4214436cf67831ea0e77dd9c2e191dcdeee81ef\": container with ID starting with 546f82ea4d394f446be68df5c4214436cf67831ea0e77dd9c2e191dcdeee81ef not found: ID does not exist" containerID="546f82ea4d394f446be68df5c4214436cf67831ea0e77dd9c2e191dcdeee81ef" Jul 06 19:29:25.320557 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:25.320485 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546f82ea4d394f446be68df5c4214436cf67831ea0e77dd9c2e191dcdeee81ef"} err="failed to get container status \"546f82ea4d394f446be68df5c4214436cf67831ea0e77dd9c2e191dcdeee81ef\": rpc error: code = NotFound desc = could not find container \"546f82ea4d394f446be68df5c4214436cf67831ea0e77dd9c2e191dcdeee81ef\": container with ID starting with 546f82ea4d394f446be68df5c4214436cf67831ea0e77dd9c2e191dcdeee81ef not found: ID does not exist" Jul 06 19:29:25.338992 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:25.338960 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-77fd76664-2s45x"] Jul 06 19:29:25.346514 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:25.346481 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-77fd76664-2s45x"] Jul 06 19:29:25.765904 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:25.765870 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97b0f20e-b183-43bc-98a7-244ca9fae1e7" path="/var/lib/kubelet/pods/97b0f20e-b183-43bc-98a7-244ca9fae1e7/volumes" Jul 06 19:29:25.933404 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:25.933360 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:26.033602 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:26.033503 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:38.415218 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:38.415114 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:38.478824 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:38.478786 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:52.457205 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:52.457170 2566 generic.go:356] "Generic (PLEG): container finished" podID="33a0983c-34cc-4244-99d0-7e3e69f94bbc" containerID="455b1728d081233ff89cfc0c56588decbbe77279901f2366201de16045c3855a" exitCode=1 Jul 06 19:29:52.457652 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:52.457245 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lf8hn" event={"ID":"33a0983c-34cc-4244-99d0-7e3e69f94bbc","Type":"ContainerDied","Data":"455b1728d081233ff89cfc0c56588decbbe77279901f2366201de16045c3855a"} Jul 06 19:29:52.457652 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:52.457574 2566 scope.go:117] "RemoveContainer" containerID="455b1728d081233ff89cfc0c56588decbbe77279901f2366201de16045c3855a" Jul 06 19:29:53.464889 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:53.464850 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lf8hn" event={"ID":"33a0983c-34cc-4244-99d0-7e3e69f94bbc","Type":"ContainerStarted","Data":"24abd13d5fb77a300e7c68b539dd631ed3e0b91dff862dda5b7bc691f2e3200b"} Jul 06 19:29:53.465386 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:29:53.465064 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-lf8hn" Jul 06 19:30:04.471580 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:04.471548 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-lf8hn" Jul 06 19:30:06.973283 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:06.973238 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:09.627317 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:09.627278 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:10.663133 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:10.663102 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:38.702256 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:38.702210 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:38.797083 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:38.797041 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c475d6674-hxsxn"] Jul 06 19:30:38.797340 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:38.797312 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-6c475d6674-hxsxn" podUID="f1ca37d7-da82-4519-88e6-7d5720e471f7" containerName="manager" containerID="cri-o://342b394b1b9879672ddb98b8606db642c24a1afd68cd0cb437da85b6ddc6d38c" gracePeriod=30 Jul 06 19:30:39.046381 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:39.046351 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c475d6674-hxsxn" Jul 06 19:30:39.189917 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:39.189879 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1ca37d7-da82-4519-88e6-7d5720e471f7-webhook-cert\") pod \"f1ca37d7-da82-4519-88e6-7d5720e471f7\" (UID: \"f1ca37d7-da82-4519-88e6-7d5720e471f7\") " Jul 06 19:30:39.190098 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:39.189957 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9qd9\" (UniqueName: \"kubernetes.io/projected/f1ca37d7-da82-4519-88e6-7d5720e471f7-kube-api-access-p9qd9\") pod \"f1ca37d7-da82-4519-88e6-7d5720e471f7\" (UID: \"f1ca37d7-da82-4519-88e6-7d5720e471f7\") " Jul 06 19:30:39.192047 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:39.192011 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1ca37d7-da82-4519-88e6-7d5720e471f7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f1ca37d7-da82-4519-88e6-7d5720e471f7" (UID: "f1ca37d7-da82-4519-88e6-7d5720e471f7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:30:39.192193 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:39.192083 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ca37d7-da82-4519-88e6-7d5720e471f7-kube-api-access-p9qd9" (OuterVolumeSpecName: "kube-api-access-p9qd9") pod "f1ca37d7-da82-4519-88e6-7d5720e471f7" (UID: "f1ca37d7-da82-4519-88e6-7d5720e471f7"). InnerVolumeSpecName "kube-api-access-p9qd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:30:39.291378 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:39.291271 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9qd9\" (UniqueName: \"kubernetes.io/projected/f1ca37d7-da82-4519-88e6-7d5720e471f7-kube-api-access-p9qd9\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:30:39.291378 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:39.291316 2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1ca37d7-da82-4519-88e6-7d5720e471f7-webhook-cert\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:30:39.717652 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:39.717621 2566 generic.go:356] "Generic (PLEG): container finished" podID="f1ca37d7-da82-4519-88e6-7d5720e471f7" containerID="342b394b1b9879672ddb98b8606db642c24a1afd68cd0cb437da85b6ddc6d38c" exitCode=0 Jul 06 19:30:39.718181 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:39.717673 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c475d6674-hxsxn" Jul 06 19:30:39.718181 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:39.717741 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c475d6674-hxsxn" event={"ID":"f1ca37d7-da82-4519-88e6-7d5720e471f7","Type":"ContainerDied","Data":"342b394b1b9879672ddb98b8606db642c24a1afd68cd0cb437da85b6ddc6d38c"} Jul 06 19:30:39.718181 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:39.717796 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c475d6674-hxsxn" event={"ID":"f1ca37d7-da82-4519-88e6-7d5720e471f7","Type":"ContainerDied","Data":"459a079803d7f63df592be4305831881e6d4da3aa6ddf9d61fd4a63f3930e34c"} Jul 06 19:30:39.718181 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:39.717818 2566 scope.go:117] "RemoveContainer" containerID="342b394b1b9879672ddb98b8606db642c24a1afd68cd0cb437da85b6ddc6d38c" Jul 06 19:30:39.726967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:39.726950 2566 scope.go:117] "RemoveContainer" containerID="342b394b1b9879672ddb98b8606db642c24a1afd68cd0cb437da85b6ddc6d38c" Jul 06 19:30:39.727214 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:30:39.727197 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"342b394b1b9879672ddb98b8606db642c24a1afd68cd0cb437da85b6ddc6d38c\": container with ID starting with 342b394b1b9879672ddb98b8606db642c24a1afd68cd0cb437da85b6ddc6d38c not found: ID does not exist" containerID="342b394b1b9879672ddb98b8606db642c24a1afd68cd0cb437da85b6ddc6d38c" Jul 06 19:30:39.727282 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:39.727222 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"342b394b1b9879672ddb98b8606db642c24a1afd68cd0cb437da85b6ddc6d38c"} err="failed to get container status \"342b394b1b9879672ddb98b8606db642c24a1afd68cd0cb437da85b6ddc6d38c\": rpc error: code = NotFound desc = could not find container \"342b394b1b9879672ddb98b8606db642c24a1afd68cd0cb437da85b6ddc6d38c\": container with ID starting with 342b394b1b9879672ddb98b8606db642c24a1afd68cd0cb437da85b6ddc6d38c not found: ID does not exist" Jul 06 19:30:39.740204 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:39.740178 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c475d6674-hxsxn"] Jul 06 19:30:39.745809 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:39.745784 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6c475d6674-hxsxn"] Jul 06 19:30:39.764103 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:39.764074 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1ca37d7-da82-4519-88e6-7d5720e471f7" path="/var/lib/kubelet/pods/f1ca37d7-da82-4519-88e6-7d5720e471f7/volumes" Jul 06 19:30:40.465242 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:40.465182 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:40.581612 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:40.581582 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c475d6674-wlhcj"] Jul 06 19:30:40.582008 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:40.581995 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97b0f20e-b183-43bc-98a7-244ca9fae1e7" containerName="authorino" Jul 06 19:30:40.582055 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:40.582010 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="97b0f20e-b183-43bc-98a7-244ca9fae1e7" containerName="authorino" Jul 06 19:30:40.582055 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:40.582020 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1ca37d7-da82-4519-88e6-7d5720e471f7" containerName="manager" Jul 06 19:30:40.582055 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:40.582025 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1ca37d7-da82-4519-88e6-7d5720e471f7" containerName="manager" Jul 06 19:30:40.582148 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:40.582084 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="97b0f20e-b183-43bc-98a7-244ca9fae1e7" containerName="authorino" Jul 06 19:30:40.582148 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:40.582094 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1ca37d7-da82-4519-88e6-7d5720e471f7" containerName="manager" Jul 06 19:30:40.585172 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:40.585155 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c475d6674-wlhcj" Jul 06 19:30:40.588763 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:40.588741 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 19:30:40.589621 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:40.589598 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 19:30:40.589621 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:40.589619 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rxfvz\"" Jul 06 19:30:40.598293 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:40.598267 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c475d6674-wlhcj"] Jul 06 19:30:40.703258 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:40.703224 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/223aa52f-6613-49da-bec7-dd08654cb587-webhook-cert\") pod \"maas-controller-6c475d6674-wlhcj\" (UID: \"223aa52f-6613-49da-bec7-dd08654cb587\") " pod="opendatahub/maas-controller-6c475d6674-wlhcj" Jul 06 19:30:40.703436 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:40.703348 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtgsw\" (UniqueName: \"kubernetes.io/projected/223aa52f-6613-49da-bec7-dd08654cb587-kube-api-access-rtgsw\") pod \"maas-controller-6c475d6674-wlhcj\" (UID: \"223aa52f-6613-49da-bec7-dd08654cb587\") " pod="opendatahub/maas-controller-6c475d6674-wlhcj" Jul 06 19:30:40.804068 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:40.803973 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtgsw\" (UniqueName: \"kubernetes.io/projected/223aa52f-6613-49da-bec7-dd08654cb587-kube-api-access-rtgsw\") pod \"maas-controller-6c475d6674-wlhcj\" (UID: \"223aa52f-6613-49da-bec7-dd08654cb587\") " pod="opendatahub/maas-controller-6c475d6674-wlhcj" Jul 06 19:30:40.804068 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:40.804026 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/223aa52f-6613-49da-bec7-dd08654cb587-webhook-cert\") pod \"maas-controller-6c475d6674-wlhcj\" (UID: \"223aa52f-6613-49da-bec7-dd08654cb587\") " pod="opendatahub/maas-controller-6c475d6674-wlhcj" Jul 06 19:30:40.806514 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:40.806485 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/223aa52f-6613-49da-bec7-dd08654cb587-webhook-cert\") pod \"maas-controller-6c475d6674-wlhcj\" (UID: \"223aa52f-6613-49da-bec7-dd08654cb587\") " pod="opendatahub/maas-controller-6c475d6674-wlhcj" Jul 06 19:30:40.813619 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:40.813598 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtgsw\" (UniqueName: \"kubernetes.io/projected/223aa52f-6613-49da-bec7-dd08654cb587-kube-api-access-rtgsw\") pod \"maas-controller-6c475d6674-wlhcj\" (UID: \"223aa52f-6613-49da-bec7-dd08654cb587\") " pod="opendatahub/maas-controller-6c475d6674-wlhcj" Jul 06 19:30:40.895636 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:40.895598 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c475d6674-wlhcj" Jul 06 19:30:41.035950 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:41.035928 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c475d6674-wlhcj"] Jul 06 19:30:41.037680 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:30:41.037642 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223aa52f_6613_49da_bec7_dd08654cb587.slice/crio-3a95895987a67e7669c2956c470a06b92b9379d71120f2350fec25566af1f0ab WatchSource:0}: Error finding container 3a95895987a67e7669c2956c470a06b92b9379d71120f2350fec25566af1f0ab: Status 404 returned error can't find the container with id 3a95895987a67e7669c2956c470a06b92b9379d71120f2350fec25566af1f0ab Jul 06 19:30:41.731989 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:41.731889 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c475d6674-wlhcj" event={"ID":"223aa52f-6613-49da-bec7-dd08654cb587","Type":"ContainerStarted","Data":"321f2173884a5b638160cfe65ff0a63327d387a94e17066c7dbed536f38d7a64"} Jul 06 19:30:41.731989 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:41.731936 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c475d6674-wlhcj" event={"ID":"223aa52f-6613-49da-bec7-dd08654cb587","Type":"ContainerStarted","Data":"3a95895987a67e7669c2956c470a06b92b9379d71120f2350fec25566af1f0ab"} Jul 06 19:30:41.732200 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:41.731988 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c475d6674-wlhcj" Jul 06 19:30:41.751444 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:41.751390 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6c475d6674-wlhcj" podStartSLOduration=1.316460931 podStartE2EDuration="1.751373249s" totalImagesPullingTime="434.912318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:30:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:30:41.74958626 +0000 UTC m=+1138.645475793" watchObservedRunningTime="2026-07-06 19:30:41.751373249 +0000 UTC m=+1138.647262781" Jul 06 19:30:52.743329 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:30:52.743299 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c475d6674-wlhcj" Jul 06 19:31:16.137683 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:31:16.137648 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:16.313296 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:31:16.313262 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:16.471679 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:31:16.471649 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:19.311886 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:31:19.311848 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:19.457313 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:31:19.457270 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:23.305425 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:31:23.305386 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:23.518843 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:31:23.518805 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:23.701269 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:31:23.701235 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:30.645411 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:31:30.645361 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:33.047470 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:31:33.047435 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:43.660197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:31:43.660162 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:31:43.691257 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:31:43.691231 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:33:31.841926 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:33:31.841891 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:43.696789 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:36:43.696762 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:36:43.723783 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:36:43.723759 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:36:57.908313 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:36:57.908279 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:14.228013 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:37:14.227935 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:14.285226 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:37:14.285191 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:20.543171 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:37:20.543137 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:43.727982 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:41:43.727954 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:41:43.755357 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:41:43.755332 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:42:24.934493 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:24.934455 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:32.695772 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:32.695740 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:40.878506 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:40.878471 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:44.376354 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:44.376319 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:46.062266 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:46.062228 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:49.219999 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:49.219959 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:49.325171 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:49.325111 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:49.697690 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:49.697656 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:56.881502 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:56.881469 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:57.154197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.154097 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:57.226101 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.226068 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t"] Jul 06 19:42:57.230387 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.230364 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.236827 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.236804 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9fb5e44b-openshift-default-dockercfg-kn7hn\"" Jul 06 19:42:57.279662 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.279630 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t"] Jul 06 19:42:57.329206 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.329175 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:57.348356 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.348324 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-token\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.348527 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.348378 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttb7r\" (UniqueName: \"kubernetes.io/projected/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-kube-api-access-ttb7r\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.348527 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.348463 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-podinfo\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.348645 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.348550 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-workload-socket\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.348645 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.348589 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-envoy\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.348645 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.348624 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-credential-socket\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.348834 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.348712 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-workload-certs\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.348834 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.348766 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-data\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.348834 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.348812 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istiod-ca-cert\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.450184 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.450152 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-workload-certs\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.450364 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.450189 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-data\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.450364 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.450218 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istiod-ca-cert\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.450364 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.450249 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-token\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.450364 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.450269 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttb7r\" (UniqueName: \"kubernetes.io/projected/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-kube-api-access-ttb7r\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.450364 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.450302 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-podinfo\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.450364 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.450335 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-workload-socket\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.450759 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.450459 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-envoy\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.450759 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.450507 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-credential-socket\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.450759 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.450619 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-data\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.450759 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.450702 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-workload-certs\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.450906 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.450852 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-workload-socket\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.450945 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.450907 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-credential-socket\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.451182 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.451165 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istiod-ca-cert\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.452848 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.452823 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-envoy\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.453001 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.452984 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-podinfo\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.462708 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.462682 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-token\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.463341 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.463324 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttb7r\" (UniqueName: \"kubernetes.io/projected/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-kube-api-access-ttb7r\") pod \"e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.541522 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.541486 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:57.697664 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.697637 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t"] Jul 06 19:42:57.699766 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:42:57.699739 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca65b3f1_69c3_4b47_8e3c_db2e6d8cc7dd.slice/crio-45935f70e804cc56b4c38a76affa03025a00cc2bdad227434d3b7df4a672deb4 WatchSource:0}: Error finding container 45935f70e804cc56b4c38a76affa03025a00cc2bdad227434d3b7df4a672deb4: Status 404 returned error can't find the container with id 45935f70e804cc56b4c38a76affa03025a00cc2bdad227434d3b7df4a672deb4 Jul 06 19:42:57.701861 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.701831 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:42:57.701927 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.701898 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:42:57.701968 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.701932 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:42:57.768156 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.768123 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" event={"ID":"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd","Type":"ContainerStarted","Data":"3d174cca6c6eb293f14e368da10febbc83caf11f0397160b57d08303d66c04fa"} Jul 06 19:42:57.768252 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.768184 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" event={"ID":"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd","Type":"ContainerStarted","Data":"45935f70e804cc56b4c38a76affa03025a00cc2bdad227434d3b7df4a672deb4"} Jul 06 19:42:57.810079 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:57.810026 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" podStartSLOduration=0.810011231 podStartE2EDuration="810.011231ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:42:57.809430015 +0000 UTC m=+1874.705319547" watchObservedRunningTime="2026-07-06 19:42:57.810011231 +0000 UTC m=+1874.705900762" Jul 06 19:42:58.403144 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:58.403114 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:58.542445 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:58.542415 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:58.547402 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:58.547376 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:58.620022 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:58.619990 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:58.773352 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:58.773315 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:58.774225 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:58.774206 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:42:58.793988 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:58.793962 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:59.509296 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:59.509261 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:59.617657 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:42:59.617625 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:01.890414 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:01.890384 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:03.606335 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:03.606299 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:03.807457 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:03.807424 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t"] Jul 06 19:43:03.807658 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:03.807611 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" podUID="ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd" containerName="istio-proxy" containerID="cri-o://3d174cca6c6eb293f14e368da10febbc83caf11f0397160b57d08303d66c04fa" gracePeriod=30 Jul 06 19:43:03.868475 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:03.868378 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:03.956646 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:03.956617 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:04.214565 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:04.214525 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:05.463497 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:05.463467 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:09.063065 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.063035 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:43:09.155629 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.155538 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-credential-socket\") pod \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " Jul 06 19:43:09.155629 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.155580 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-podinfo\") pod \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " Jul 06 19:43:09.155629 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.155617 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttb7r\" (UniqueName: \"kubernetes.io/projected/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-kube-api-access-ttb7r\") pod \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " Jul 06 19:43:09.155953 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.155639 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-envoy\") pod \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " Jul 06 19:43:09.155953 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.155661 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-workload-certs\") pod \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " Jul 06 19:43:09.155953 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.155708 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istiod-ca-cert\") pod \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " Jul 06 19:43:09.155953 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.155814 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-token\") pod \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " Jul 06 19:43:09.155953 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.155839 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-workload-socket\") pod \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " Jul 06 19:43:09.155953 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.155883 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-data\") pod \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\" (UID: \"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd\") " Jul 06 19:43:09.155953 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.155880 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd" (UID: "ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:09.156296 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.155984 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd" (UID: "ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:09.156296 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.156114 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd" (UID: "ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:09.156296 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.156190 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd" (UID: "ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:09.156296 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.156204 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-workload-socket\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:09.156296 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.156227 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-credential-socket\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:09.156296 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.156243 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-workload-certs\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:09.156526 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.156381 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd" (UID: "ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:43:09.157923 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.157898 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd" (UID: "ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:43:09.158015 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.157953 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd" (UID: "ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:09.158216 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.158200 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd" (UID: "ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:43:09.158260 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.158235 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-kube-api-access-ttb7r" (OuterVolumeSpecName: "kube-api-access-ttb7r") pod "ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd" (UID: "ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd"). InnerVolumeSpecName "kube-api-access-ttb7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:43:09.257029 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.256991 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-token\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:09.257029 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.257022 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-data\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:09.257029 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.257033 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-podinfo\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:09.257278 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.257046 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttb7r\" (UniqueName: \"kubernetes.io/projected/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-kube-api-access-ttb7r\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:09.257278 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.257059 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istio-envoy\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:09.257278 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.257070 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd-istiod-ca-cert\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:09.835013 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.834979 2566 generic.go:356] "Generic (PLEG): container finished" podID="ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd" containerID="3d174cca6c6eb293f14e368da10febbc83caf11f0397160b57d08303d66c04fa" exitCode=0 Jul 06 19:43:09.835207 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.835035 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" Jul 06 19:43:09.835207 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.835060 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" event={"ID":"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd","Type":"ContainerDied","Data":"3d174cca6c6eb293f14e368da10febbc83caf11f0397160b57d08303d66c04fa"} Jul 06 19:43:09.835207 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.835111 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t" event={"ID":"ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd","Type":"ContainerDied","Data":"45935f70e804cc56b4c38a76affa03025a00cc2bdad227434d3b7df4a672deb4"} Jul 06 19:43:09.835207 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.835131 2566 scope.go:117] "RemoveContainer" containerID="3d174cca6c6eb293f14e368da10febbc83caf11f0397160b57d08303d66c04fa" Jul 06 19:43:09.844375 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.844363 2566 scope.go:117] "RemoveContainer" containerID="3d174cca6c6eb293f14e368da10febbc83caf11f0397160b57d08303d66c04fa" Jul 06 19:43:09.844621 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:43:09.844605 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d174cca6c6eb293f14e368da10febbc83caf11f0397160b57d08303d66c04fa\": container with ID starting with 3d174cca6c6eb293f14e368da10febbc83caf11f0397160b57d08303d66c04fa not found: ID does not exist" containerID="3d174cca6c6eb293f14e368da10febbc83caf11f0397160b57d08303d66c04fa" Jul 06 19:43:09.844674 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.844630 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d174cca6c6eb293f14e368da10febbc83caf11f0397160b57d08303d66c04fa"} err="failed to get container status \"3d174cca6c6eb293f14e368da10febbc83caf11f0397160b57d08303d66c04fa\": rpc error: code = NotFound desc = could not find container \"3d174cca6c6eb293f14e368da10febbc83caf11f0397160b57d08303d66c04fa\": container with ID starting with 3d174cca6c6eb293f14e368da10febbc83caf11f0397160b57d08303d66c04fa not found: ID does not exist" Jul 06 19:43:09.856588 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.856563 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t"] Jul 06 19:43:09.861576 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:09.861549 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9fb5e44b-openshift-default-6d969f5fc-6556t"] Jul 06 19:43:10.308685 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:10.308652 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:10.464690 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:10.464660 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:10.879236 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:10.879194 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr"] Jul 06 19:43:10.879697 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:10.879680 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd" containerName="istio-proxy" Jul 06 19:43:10.879825 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:10.879700 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd" containerName="istio-proxy" Jul 06 19:43:10.879825 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:10.879803 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd" containerName="istio-proxy" Jul 06 19:43:10.884222 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:10.884199 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:10.889226 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:10.889206 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-557aa455-openshift-default-dockercfg-qb7gd\"" Jul 06 19:43:10.913750 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:10.913704 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr"] Jul 06 19:43:10.970952 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:10.970923 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-credential-socket\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:10.971326 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:10.971301 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60907eb2-6143-4977-a909-fd8f245b73b6-istiod-ca-cert\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:10.971701 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:10.971616 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60907eb2-6143-4977-a909-fd8f245b73b6-istio-token\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:10.972103 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:10.972083 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqdvf\" (UniqueName: \"kubernetes.io/projected/60907eb2-6143-4977-a909-fd8f245b73b6-kube-api-access-kqdvf\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:10.972419 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:10.972402 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-workload-socket\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:10.972546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:10.972531 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-istio-envoy\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:10.972712 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:10.972689 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60907eb2-6143-4977-a909-fd8f245b73b6-istio-podinfo\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:10.972712 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:10.972750 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-workload-certs\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:10.972712 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:10.972765 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:10.972712 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:10.972811 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-istio-data\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:11.073483 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.073392 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-credential-socket\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:11.073483 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.073429 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60907eb2-6143-4977-a909-fd8f245b73b6-istiod-ca-cert\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:11.073483 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.073467 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60907eb2-6143-4977-a909-fd8f245b73b6-istio-token\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:11.074168 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.073502 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqdvf\" (UniqueName: \"kubernetes.io/projected/60907eb2-6143-4977-a909-fd8f245b73b6-kube-api-access-kqdvf\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:11.074168 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.073542 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-workload-socket\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:11.074168 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.073572 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-istio-envoy\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:11.074168 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.073624 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60907eb2-6143-4977-a909-fd8f245b73b6-istio-podinfo\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:11.074168 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.073662 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-workload-certs\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:11.074168 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.073690 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-istio-data\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:11.074168 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.073961 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-credential-socket\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:11.074168 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.074073 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-workload-socket\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:11.074168 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.074100 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-workload-certs\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:11.074488 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.074174 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-istio-data\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:11.074488 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.074246 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60907eb2-6143-4977-a909-fd8f245b73b6-istiod-ca-cert\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:11.075852 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.075829 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-istio-envoy\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:11.076224 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.076203 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60907eb2-6143-4977-a909-fd8f245b73b6-istio-podinfo\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:11.095121 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.095094 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqdvf\" (UniqueName: \"kubernetes.io/projected/60907eb2-6143-4977-a909-fd8f245b73b6-kube-api-access-kqdvf\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:11.096186 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.096160 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60907eb2-6143-4977-a909-fd8f245b73b6-istio-token\") pod \"e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:11.181693 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.181658 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:11.194680 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.194651 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:11.360379 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.360346 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr"] Jul 06 19:43:11.361616 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:43:11.361587 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60907eb2_6143_4977_a909_fd8f245b73b6.slice/crio-fc81ec2eba1433013144aebd7a4f89a6e5e1049ca063d6c01222fd37ea86058b WatchSource:0}: Error finding container fc81ec2eba1433013144aebd7a4f89a6e5e1049ca063d6c01222fd37ea86058b: Status 404 returned error can't find the container with id fc81ec2eba1433013144aebd7a4f89a6e5e1049ca063d6c01222fd37ea86058b Jul 06 19:43:11.363835 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.363801 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:43:11.363947 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.363882 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:43:11.363947 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.363932 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:43:11.763810 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.763767 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd" path="/var/lib/kubelet/pods/ca65b3f1-69c3-4b47-8e3c-db2e6d8cc7dd/volumes" Jul 06 19:43:11.848362 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.848317 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" event={"ID":"60907eb2-6143-4977-a909-fd8f245b73b6","Type":"ContainerStarted","Data":"33d6c960177d35286bd778756ea8a7e3d68977e5c6a01b34a9b67ab68ecdbfc0"} Jul 06 19:43:11.848548 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.848389 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" event={"ID":"60907eb2-6143-4977-a909-fd8f245b73b6","Type":"ContainerStarted","Data":"fc81ec2eba1433013144aebd7a4f89a6e5e1049ca063d6c01222fd37ea86058b"} Jul 06 19:43:11.916800 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.916753 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:11.935181 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:11.935116 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" podStartSLOduration=1.935092308 podStartE2EDuration="1.935092308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:43:11.927203162 +0000 UTC m=+1888.823092726" watchObservedRunningTime="2026-07-06 19:43:11.935092308 +0000 UTC m=+1888.830981841" Jul 06 19:43:12.164941 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:12.164898 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:12.195555 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:12.195521 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:12.200758 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:12.200716 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:12.445228 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:12.445114 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:12.855292 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:12.855207 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:12.856080 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:12.856064 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:13.191299 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:13.191263 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:13.356201 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:13.356167 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:14.232044 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:14.232007 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:17.024870 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:17.024837 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:17.572378 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:17.572304 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:23.207811 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:23.207773 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr"] Jul 06 19:43:23.208201 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:23.208028 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" podUID="60907eb2-6143-4977-a909-fd8f245b73b6" containerName="istio-proxy" containerID="cri-o://33d6c960177d35286bd778756ea8a7e3d68977e5c6a01b34a9b67ab68ecdbfc0" gracePeriod=30 Jul 06 19:43:23.250931 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:23.250892 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:23.374129 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:23.374099 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:24.724745 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:24.724701 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:27.858307 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:27.858257 2566 patch_prober.go:28] interesting pod/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 19:43:27.858745 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:27.858320 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" podUID="60907eb2-6143-4977-a909-fd8f245b73b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:43:28.474750 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.474672 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:28.632884 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.632850 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-istio-envoy\") pod \"60907eb2-6143-4977-a909-fd8f245b73b6\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " Jul 06 19:43:28.633083 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.632901 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-workload-socket\") pod \"60907eb2-6143-4977-a909-fd8f245b73b6\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " Jul 06 19:43:28.633083 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.632922 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqdvf\" (UniqueName: \"kubernetes.io/projected/60907eb2-6143-4977-a909-fd8f245b73b6-kube-api-access-kqdvf\") pod \"60907eb2-6143-4977-a909-fd8f245b73b6\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " Jul 06 19:43:28.633083 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.632951 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60907eb2-6143-4977-a909-fd8f245b73b6-istio-token\") pod \"60907eb2-6143-4977-a909-fd8f245b73b6\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " Jul 06 19:43:28.633083 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.632984 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60907eb2-6143-4977-a909-fd8f245b73b6-istiod-ca-cert\") pod \"60907eb2-6143-4977-a909-fd8f245b73b6\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " Jul 06 19:43:28.633083 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.633026 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-workload-certs\") pod \"60907eb2-6143-4977-a909-fd8f245b73b6\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " Jul 06 19:43:28.633083 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.633077 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-credential-socket\") pod \"60907eb2-6143-4977-a909-fd8f245b73b6\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " Jul 06 19:43:28.633410 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.633104 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60907eb2-6143-4977-a909-fd8f245b73b6-istio-podinfo\") pod \"60907eb2-6143-4977-a909-fd8f245b73b6\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " Jul 06 19:43:28.633410 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.633179 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-istio-data\") pod \"60907eb2-6143-4977-a909-fd8f245b73b6\" (UID: \"60907eb2-6143-4977-a909-fd8f245b73b6\") " Jul 06 19:43:28.633410 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.633187 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "60907eb2-6143-4977-a909-fd8f245b73b6" (UID: "60907eb2-6143-4977-a909-fd8f245b73b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:28.633572 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.633476 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-workload-socket\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:28.633572 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.633553 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "60907eb2-6143-4977-a909-fd8f245b73b6" (UID: "60907eb2-6143-4977-a909-fd8f245b73b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:28.633694 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.633589 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60907eb2-6143-4977-a909-fd8f245b73b6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "60907eb2-6143-4977-a909-fd8f245b73b6" (UID: "60907eb2-6143-4977-a909-fd8f245b73b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:43:28.633694 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.633666 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "60907eb2-6143-4977-a909-fd8f245b73b6" (UID: "60907eb2-6143-4977-a909-fd8f245b73b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:28.633694 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.633679 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-istio-data" (OuterVolumeSpecName: "istio-data") pod "60907eb2-6143-4977-a909-fd8f245b73b6" (UID: "60907eb2-6143-4977-a909-fd8f245b73b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:28.635260 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.635234 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "60907eb2-6143-4977-a909-fd8f245b73b6" (UID: "60907eb2-6143-4977-a909-fd8f245b73b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:28.635260 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.635253 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60907eb2-6143-4977-a909-fd8f245b73b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "60907eb2-6143-4977-a909-fd8f245b73b6" (UID: "60907eb2-6143-4977-a909-fd8f245b73b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:43:28.635419 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.635302 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60907eb2-6143-4977-a909-fd8f245b73b6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "60907eb2-6143-4977-a909-fd8f245b73b6" (UID: "60907eb2-6143-4977-a909-fd8f245b73b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:43:28.635419 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.635310 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60907eb2-6143-4977-a909-fd8f245b73b6-kube-api-access-kqdvf" (OuterVolumeSpecName: "kube-api-access-kqdvf") pod "60907eb2-6143-4977-a909-fd8f245b73b6" (UID: "60907eb2-6143-4977-a909-fd8f245b73b6"). InnerVolumeSpecName "kube-api-access-kqdvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:43:28.734838 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.734749 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60907eb2-6143-4977-a909-fd8f245b73b6-istio-token\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:28.734838 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.734776 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60907eb2-6143-4977-a909-fd8f245b73b6-istiod-ca-cert\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:28.734838 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.734786 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-workload-certs\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:28.734838 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.734794 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-credential-socket\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:28.734838 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.734802 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60907eb2-6143-4977-a909-fd8f245b73b6-istio-podinfo\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:28.734838 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.734813 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-istio-data\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:28.734838 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.734820 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60907eb2-6143-4977-a909-fd8f245b73b6-istio-envoy\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:28.734838 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.734828 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqdvf\" (UniqueName: \"kubernetes.io/projected/60907eb2-6143-4977-a909-fd8f245b73b6-kube-api-access-kqdvf\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:28.942255 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.942220 2566 generic.go:356] "Generic (PLEG): container finished" podID="60907eb2-6143-4977-a909-fd8f245b73b6" containerID="33d6c960177d35286bd778756ea8a7e3d68977e5c6a01b34a9b67ab68ecdbfc0" exitCode=0 Jul 06 19:43:28.942683 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.942296 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" Jul 06 19:43:28.942683 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.942299 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" event={"ID":"60907eb2-6143-4977-a909-fd8f245b73b6","Type":"ContainerDied","Data":"33d6c960177d35286bd778756ea8a7e3d68977e5c6a01b34a9b67ab68ecdbfc0"} Jul 06 19:43:28.942683 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.942390 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr" event={"ID":"60907eb2-6143-4977-a909-fd8f245b73b6","Type":"ContainerDied","Data":"fc81ec2eba1433013144aebd7a4f89a6e5e1049ca063d6c01222fd37ea86058b"} Jul 06 19:43:28.942683 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.942422 2566 scope.go:117] "RemoveContainer" containerID="33d6c960177d35286bd778756ea8a7e3d68977e5c6a01b34a9b67ab68ecdbfc0" Jul 06 19:43:28.954373 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.954346 2566 scope.go:117] "RemoveContainer" containerID="33d6c960177d35286bd778756ea8a7e3d68977e5c6a01b34a9b67ab68ecdbfc0" Jul 06 19:43:28.954868 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:43:28.954842 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33d6c960177d35286bd778756ea8a7e3d68977e5c6a01b34a9b67ab68ecdbfc0\": container with ID starting with 33d6c960177d35286bd778756ea8a7e3d68977e5c6a01b34a9b67ab68ecdbfc0 not found: ID does not exist" containerID="33d6c960177d35286bd778756ea8a7e3d68977e5c6a01b34a9b67ab68ecdbfc0" Jul 06 19:43:28.954941 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.954881 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33d6c960177d35286bd778756ea8a7e3d68977e5c6a01b34a9b67ab68ecdbfc0"} err="failed to get container status \"33d6c960177d35286bd778756ea8a7e3d68977e5c6a01b34a9b67ab68ecdbfc0\": rpc error: code = NotFound desc = could not find container \"33d6c960177d35286bd778756ea8a7e3d68977e5c6a01b34a9b67ab68ecdbfc0\": container with ID starting with 33d6c960177d35286bd778756ea8a7e3d68977e5c6a01b34a9b67ab68ecdbfc0 not found: ID does not exist" Jul 06 19:43:28.969199 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.969169 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr"] Jul 06 19:43:28.971160 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:28.971140 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-557aa455-openshift-default-6b8b6b8d49-hxgzr"] Jul 06 19:43:29.675218 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:29.675186 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:29.763910 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:29.763878 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60907eb2-6143-4977-a909-fd8f245b73b6" path="/var/lib/kubelet/pods/60907eb2-6143-4977-a909-fd8f245b73b6/volumes" Jul 06 19:43:29.775934 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:29.775893 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:29.905999 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:29.905968 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm"] Jul 06 19:43:29.906583 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:29.906564 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60907eb2-6143-4977-a909-fd8f245b73b6" containerName="istio-proxy" Jul 06 19:43:29.906631 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:29.906590 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="60907eb2-6143-4977-a909-fd8f245b73b6" containerName="istio-proxy" Jul 06 19:43:29.906707 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:29.906695 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="60907eb2-6143-4977-a909-fd8f245b73b6" containerName="istio-proxy" Jul 06 19:43:29.911594 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:29.911574 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:29.914833 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:29.914815 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-cb88f886-openshift-default-dockercfg-8drm9\"" Jul 06 19:43:29.926289 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:29.926241 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm"] Jul 06 19:43:29.943481 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:29.943460 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-istio-data\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:29.943877 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:29.943510 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-workload-socket\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:29.943877 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:29.943543 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqmb6\" (UniqueName: \"kubernetes.io/projected/91638c73-4948-4697-b36c-6ace36ae5844-kube-api-access-tqmb6\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:29.943877 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:29.943594 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-istio-envoy\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:29.943877 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:29.943619 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91638c73-4948-4697-b36c-6ace36ae5844-istio-token\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:29.943877 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:29.943644 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-workload-certs\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:29.943877 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:29.943667 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91638c73-4948-4697-b36c-6ace36ae5844-istio-podinfo\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:29.943877 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:29.943702 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-credential-socket\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:29.943877 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:29.943761 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91638c73-4948-4697-b36c-6ace36ae5844-istiod-ca-cert\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:29.978228 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:29.978175 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:30.045053 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.045020 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-workload-socket\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:30.045220 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.045058 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqmb6\" (UniqueName: \"kubernetes.io/projected/91638c73-4948-4697-b36c-6ace36ae5844-kube-api-access-tqmb6\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:30.045220 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.045110 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-istio-envoy\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:30.045220 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.045132 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91638c73-4948-4697-b36c-6ace36ae5844-istio-token\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:30.045220 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.045156 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-workload-certs\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:30.045428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.045285 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91638c73-4948-4697-b36c-6ace36ae5844-istio-podinfo\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:30.045428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.045331 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-credential-socket\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:30.045428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.045380 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91638c73-4948-4697-b36c-6ace36ae5844-istiod-ca-cert\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:30.045634 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.045460 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-istio-data\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:30.045634 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.045480 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-workload-socket\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:30.045771 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.045635 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-workload-certs\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:30.045771 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.045675 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-credential-socket\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:30.045889 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.045841 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-istio-data\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:30.046135 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.046117 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91638c73-4948-4697-b36c-6ace36ae5844-istiod-ca-cert\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:30.047443 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.047425 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-istio-envoy\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:30.047777 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.047756 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91638c73-4948-4697-b36c-6ace36ae5844-istio-podinfo\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:30.067858 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.067830 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqmb6\" (UniqueName: \"kubernetes.io/projected/91638c73-4948-4697-b36c-6ace36ae5844-kube-api-access-tqmb6\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:30.075610 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.075585 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91638c73-4948-4697-b36c-6ace36ae5844-istio-token\") pod \"e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:30.114845 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.114798 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:30.222348 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.222234 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:30.366041 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.366016 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm"] Jul 06 19:43:30.368120 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:43:30.368091 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91638c73_4948_4697_b36c_6ace36ae5844.slice/crio-7d70dd0a876028730f1c1dc064e5f2dd6bb5d7afb8a5dabc33d9dfc3db34295b WatchSource:0}: Error finding container 7d70dd0a876028730f1c1dc064e5f2dd6bb5d7afb8a5dabc33d9dfc3db34295b: Status 404 returned error can't find the container with id 7d70dd0a876028730f1c1dc064e5f2dd6bb5d7afb8a5dabc33d9dfc3db34295b Jul 06 19:43:30.370331 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.370299 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:43:30.370413 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.370364 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:43:30.370413 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.370393 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:43:30.956832 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.956794 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" event={"ID":"91638c73-4948-4697-b36c-6ace36ae5844","Type":"ContainerStarted","Data":"db8b2eb75f16b05759fcce0f0aed7977ebf7dfae079beb504e1cffacdabd3fbe"} Jul 06 19:43:30.957315 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.956854 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" event={"ID":"91638c73-4948-4697-b36c-6ace36ae5844","Type":"ContainerStarted","Data":"7d70dd0a876028730f1c1dc064e5f2dd6bb5d7afb8a5dabc33d9dfc3db34295b"} Jul 06 19:43:30.989973 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:30.989921 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" podStartSLOduration=1.989902411 podStartE2EDuration="1.989902411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:43:30.989356248 +0000 UTC m=+1907.885245813" watchObservedRunningTime="2026-07-06 19:43:30.989902411 +0000 UTC m=+1907.885791943" Jul 06 19:43:31.222591 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:31.222503 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:31.227546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:31.227518 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:31.311822 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:31.311785 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:31.536291 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:31.536160 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:31.704610 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:31.704578 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:31.962694 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:31.962659 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:31.963822 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:31.963802 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:32.380022 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:32.379934 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:32.591873 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:32.591839 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:33.806473 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:33.806441 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:35.612228 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:35.612190 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:35.820415 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:35.820381 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm"] Jul 06 19:43:35.820597 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:35.820566 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" podUID="91638c73-4948-4697-b36c-6ace36ae5844" containerName="istio-proxy" containerID="cri-o://db8b2eb75f16b05759fcce0f0aed7977ebf7dfae079beb504e1cffacdabd3fbe" gracePeriod=30 Jul 06 19:43:35.870026 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:35.869940 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:35.978788 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:35.978716 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:36.233489 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:36.233457 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:37.198795 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:37.198766 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:41.019937 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.019896 2566 generic.go:356] "Generic (PLEG): container finished" podID="91638c73-4948-4697-b36c-6ace36ae5844" containerID="db8b2eb75f16b05759fcce0f0aed7977ebf7dfae079beb504e1cffacdabd3fbe" exitCode=0 Jul 06 19:43:41.020318 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.019968 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" event={"ID":"91638c73-4948-4697-b36c-6ace36ae5844","Type":"ContainerDied","Data":"db8b2eb75f16b05759fcce0f0aed7977ebf7dfae079beb504e1cffacdabd3fbe"} Jul 06 19:43:41.076610 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.076579 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:41.150162 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.150070 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-istio-envoy\") pod \"91638c73-4948-4697-b36c-6ace36ae5844\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " Jul 06 19:43:41.150335 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.150209 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-credential-socket\") pod \"91638c73-4948-4697-b36c-6ace36ae5844\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " Jul 06 19:43:41.150335 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.150276 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91638c73-4948-4697-b36c-6ace36ae5844-istiod-ca-cert\") pod \"91638c73-4948-4697-b36c-6ace36ae5844\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " Jul 06 19:43:41.150335 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.150317 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-workload-socket\") pod \"91638c73-4948-4697-b36c-6ace36ae5844\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " Jul 06 19:43:41.150504 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.150341 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91638c73-4948-4697-b36c-6ace36ae5844-istio-podinfo\") pod \"91638c73-4948-4697-b36c-6ace36ae5844\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " Jul 06 19:43:41.150504 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.150385 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-workload-certs\") pod \"91638c73-4948-4697-b36c-6ace36ae5844\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " Jul 06 19:43:41.150504 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.150422 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-istio-data\") pod \"91638c73-4948-4697-b36c-6ace36ae5844\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " Jul 06 19:43:41.150504 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.150474 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqmb6\" (UniqueName: \"kubernetes.io/projected/91638c73-4948-4697-b36c-6ace36ae5844-kube-api-access-tqmb6\") pod \"91638c73-4948-4697-b36c-6ace36ae5844\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " Jul 06 19:43:41.150504 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.150487 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "91638c73-4948-4697-b36c-6ace36ae5844" (UID: "91638c73-4948-4697-b36c-6ace36ae5844"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:41.150781 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.150531 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91638c73-4948-4697-b36c-6ace36ae5844-istio-token\") pod \"91638c73-4948-4697-b36c-6ace36ae5844\" (UID: \"91638c73-4948-4697-b36c-6ace36ae5844\") " Jul 06 19:43:41.150781 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.150648 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91638c73-4948-4697-b36c-6ace36ae5844-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "91638c73-4948-4697-b36c-6ace36ae5844" (UID: "91638c73-4948-4697-b36c-6ace36ae5844"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:43:41.150781 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.150625 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "91638c73-4948-4697-b36c-6ace36ae5844" (UID: "91638c73-4948-4697-b36c-6ace36ae5844"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:41.150781 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.150702 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-istio-data" (OuterVolumeSpecName: "istio-data") pod "91638c73-4948-4697-b36c-6ace36ae5844" (UID: "91638c73-4948-4697-b36c-6ace36ae5844"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:41.150781 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.150737 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "91638c73-4948-4697-b36c-6ace36ae5844" (UID: "91638c73-4948-4697-b36c-6ace36ae5844"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:41.151026 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.150873 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-credential-socket\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:41.151026 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.150899 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91638c73-4948-4697-b36c-6ace36ae5844-istiod-ca-cert\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:41.151026 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.150914 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-workload-socket\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:41.151026 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.150926 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-workload-certs\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:41.151026 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.150938 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-istio-data\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:41.152251 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.152224 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "91638c73-4948-4697-b36c-6ace36ae5844" (UID: "91638c73-4948-4697-b36c-6ace36ae5844"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:41.152588 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.152569 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91638c73-4948-4697-b36c-6ace36ae5844-kube-api-access-tqmb6" (OuterVolumeSpecName: "kube-api-access-tqmb6") pod "91638c73-4948-4697-b36c-6ace36ae5844" (UID: "91638c73-4948-4697-b36c-6ace36ae5844"). InnerVolumeSpecName "kube-api-access-tqmb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:43:41.153019 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.152995 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91638c73-4948-4697-b36c-6ace36ae5844-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "91638c73-4948-4697-b36c-6ace36ae5844" (UID: "91638c73-4948-4697-b36c-6ace36ae5844"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:43:41.153104 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.153003 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91638c73-4948-4697-b36c-6ace36ae5844-istio-token" (OuterVolumeSpecName: "istio-token") pod "91638c73-4948-4697-b36c-6ace36ae5844" (UID: "91638c73-4948-4697-b36c-6ace36ae5844"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:43:41.251517 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.251469 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91638c73-4948-4697-b36c-6ace36ae5844-istio-podinfo\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:41.251517 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.251511 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqmb6\" (UniqueName: \"kubernetes.io/projected/91638c73-4948-4697-b36c-6ace36ae5844-kube-api-access-tqmb6\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:41.251517 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.251521 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91638c73-4948-4697-b36c-6ace36ae5844-istio-token\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:41.251517 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:41.251529 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91638c73-4948-4697-b36c-6ace36ae5844-istio-envoy\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:43:42.026079 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:42.025994 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" Jul 06 19:43:42.026079 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:42.026029 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm" event={"ID":"91638c73-4948-4697-b36c-6ace36ae5844","Type":"ContainerDied","Data":"7d70dd0a876028730f1c1dc064e5f2dd6bb5d7afb8a5dabc33d9dfc3db34295b"} Jul 06 19:43:42.026521 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:42.026091 2566 scope.go:117] "RemoveContainer" containerID="db8b2eb75f16b05759fcce0f0aed7977ebf7dfae079beb504e1cffacdabd3fbe" Jul 06 19:43:42.048300 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:42.048272 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm"] Jul 06 19:43:42.055677 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:42.055653 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-cb88f886-openshift-default-6b4f67f8d7-w6qrm"] Jul 06 19:43:42.360506 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:42.360412 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:42.467975 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:42.467941 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:42.904613 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:42.904581 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:43.155379 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:43.155296 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:43.764936 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:43.764903 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91638c73-4948-4697-b36c-6ace36ae5844" path="/var/lib/kubelet/pods/91638c73-4948-4697-b36c-6ace36ae5844/volumes" Jul 06 19:43:44.840329 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:44.840291 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:51.716380 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:51.716344 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:51.901617 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:51.901585 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:52.030127 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:52.030023 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:52.635065 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:43:52.635030 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:18.718071 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:44:18.718038 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:25.451874 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:44:25.451837 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:25.609524 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:44:25.609485 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:25.738409 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:44:25.738325 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:41.738880 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:44:41.738794 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:48.339508 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:44:48.339456 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:48.460267 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:44:48.460232 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:48.655100 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:44:48.655006 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:59.204732 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:44:59.204685 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:00.080353 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:00.080314 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:06.536288 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:06.536252 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:06.688192 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:06.688156 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:06.925878 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:06.925840 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:07.133625 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:07.133593 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:07.833602 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:07.833571 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:07.938005 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:07.937973 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:09.773498 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:09.773469 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:16.519634 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:16.519599 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:16.645397 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:16.645364 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:17.328107 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:17.328072 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:23.825499 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:23.825462 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:23.927181 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:23.927125 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:24.129174 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.129055 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:24.531437 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.531400 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4"] Jul 06 19:45:24.531988 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.531972 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91638c73-4948-4697-b36c-6ace36ae5844" containerName="istio-proxy" Jul 06 19:45:24.532093 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.531990 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="91638c73-4948-4697-b36c-6ace36ae5844" containerName="istio-proxy" Jul 06 19:45:24.532155 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.532099 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="91638c73-4948-4697-b36c-6ace36ae5844" containerName="istio-proxy" Jul 06 19:45:24.535580 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.535559 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.537608 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.537571 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:24.539580 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:45:24.539559 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" is forbidden: User \"system:node:ip-10-0-133-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-201.ec2.internal' and this object" podUID="e68641b6-f8ad-427e-a7e9-2730513420dd" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.539698 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:45:24.539593 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-mt-b3c8385f-openshift-default-dockercfg-fnlww\" is forbidden: User \"system:node:ip-10-0-133-201.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-201.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"e2e-mt-b3c8385f-openshift-default-dockercfg-fnlww\"" type="*v1.Secret" Jul 06 19:45:24.552633 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.552609 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4"] Jul 06 19:45:24.598538 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.598504 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctpsh\" (UniqueName: \"kubernetes.io/projected/e68641b6-f8ad-427e-a7e9-2730513420dd-kube-api-access-ctpsh\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.598746 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.598558 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68641b6-f8ad-427e-a7e9-2730513420dd-istiod-ca-cert\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.598746 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.598595 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-workload-socket\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.598746 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.598658 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-workload-certs\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.598746 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.598687 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-podinfo\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.598746 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.598740 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-token\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.599018 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.598766 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-credential-socket\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.599018 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.598782 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-envoy\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.599018 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.598806 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-data\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.700035 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.699992 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68641b6-f8ad-427e-a7e9-2730513420dd-istiod-ca-cert\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.700216 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.700056 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-workload-socket\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.700216 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.700107 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-workload-certs\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.700216 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.700146 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-podinfo\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.700216 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.700178 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-token\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.700436 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.700223 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-credential-socket\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.700436 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.700247 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-envoy\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.700436 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.700271 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-data\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.700436 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.700316 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctpsh\" (UniqueName: \"kubernetes.io/projected/e68641b6-f8ad-427e-a7e9-2730513420dd-kube-api-access-ctpsh\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.700649 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.700590 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-workload-socket\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.700707 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.700662 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-credential-socket\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.700707 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.700689 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-workload-certs\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.700881 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.700751 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-data\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.700948 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.700913 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68641b6-f8ad-427e-a7e9-2730513420dd-istiod-ca-cert\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.702617 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.702597 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-envoy\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.702981 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.702962 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-podinfo\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.714399 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.714368 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-token\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.714490 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.714443 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctpsh\" (UniqueName: \"kubernetes.io/projected/e68641b6-f8ad-427e-a7e9-2730513420dd-kube-api-access-ctpsh\") pod \"e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:24.746240 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:24.746204 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:25.505741 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:25.505684 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b3c8385f-openshift-default-dockercfg-fnlww\"" Jul 06 19:45:25.506744 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:25.506701 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:25.666699 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:25.666675 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4"] Jul 06 19:45:25.669165 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:45:25.669138 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68641b6_f8ad_427e_a7e9_2730513420dd.slice/crio-93d458c8cc31dd42a689f60fe5ff5d9878294864e572fed7945364549ff431e5 WatchSource:0}: Error finding container 93d458c8cc31dd42a689f60fe5ff5d9878294864e572fed7945364549ff431e5: Status 404 returned error can't find the container with id 93d458c8cc31dd42a689f60fe5ff5d9878294864e572fed7945364549ff431e5 Jul 06 19:45:25.671268 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:25.671233 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:45:25.671372 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:25.671298 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:45:25.671372 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:25.671328 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:45:26.585607 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:26.585571 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" event={"ID":"e68641b6-f8ad-427e-a7e9-2730513420dd","Type":"ContainerStarted","Data":"452f0ad83138cbc0c2f1f463bdf3462cc2606603ee26cfc251c9507452378aa4"} Jul 06 19:45:26.586019 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:26.585631 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" event={"ID":"e68641b6-f8ad-427e-a7e9-2730513420dd","Type":"ContainerStarted","Data":"93d458c8cc31dd42a689f60fe5ff5d9878294864e572fed7945364549ff431e5"} Jul 06 19:45:26.625385 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:26.625335 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" podStartSLOduration=2.62531806 podStartE2EDuration="2.62531806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:45:26.623121915 +0000 UTC m=+2023.519011449" watchObservedRunningTime="2026-07-06 19:45:26.62531806 +0000 UTC m=+2023.521207592" Jul 06 19:45:27.507916 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:27.507882 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:27.512747 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:27.512699 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:27.591066 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:27.591036 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:27.592122 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:27.592097 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:28.870083 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:28.870048 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:29.017573 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:29.017542 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:29.158133 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:29.158098 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:29.644952 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:29.644851 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:30.087513 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:30.087477 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:30.190136 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:30.190100 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:33.230161 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:33.230125 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:33.923169 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:33.923127 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:34.038442 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:34.038409 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:34.463576 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:34.463504 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:40.669976 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:40.669949 2566 generic.go:356] "Generic (PLEG): container finished" podID="e68641b6-f8ad-427e-a7e9-2730513420dd" containerID="452f0ad83138cbc0c2f1f463bdf3462cc2606603ee26cfc251c9507452378aa4" exitCode=0 Jul 06 19:45:40.670398 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:40.670005 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" event={"ID":"e68641b6-f8ad-427e-a7e9-2730513420dd","Type":"ContainerDied","Data":"452f0ad83138cbc0c2f1f463bdf3462cc2606603ee26cfc251c9507452378aa4"} Jul 06 19:45:40.670398 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:40.670377 2566 scope.go:117] "RemoveContainer" containerID="452f0ad83138cbc0c2f1f463bdf3462cc2606603ee26cfc251c9507452378aa4" Jul 06 19:45:40.671423 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:40.671389 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:45:40.671537 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:40.671476 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:45:40.671537 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:40.671526 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:45:41.300162 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:41.300131 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:41.441388 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:41.441351 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:41.678736 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:41.678687 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" event={"ID":"e68641b6-f8ad-427e-a7e9-2730513420dd","Type":"ContainerStarted","Data":"7e01a6cecc43503330fe6f44d190f3d000240299c1c474ed97a13a8623481e0a"} Jul 06 19:45:41.787839 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:41.787807 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:42.187400 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:42.187367 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:42.341598 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:42.341564 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:42.508017 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:42.507915 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:42.509102 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:42.509079 2566 patch_prober.go:28] interesting pod/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 06 19:45:42.509164 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:42.509143 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" podUID="e68641b6-f8ad-427e-a7e9-2730513420dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 06 19:45:42.591410 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:42.591377 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:43.507586 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:43.507552 2566 patch_prober.go:28] interesting pod/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 06 19:45:43.508021 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:43.507601 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" podUID="e68641b6-f8ad-427e-a7e9-2730513420dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 06 19:45:44.507484 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:44.507452 2566 patch_prober.go:28] interesting pod/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 06 19:45:44.507668 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:44.507515 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" podUID="e68641b6-f8ad-427e-a7e9-2730513420dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 06 19:45:45.507877 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:45.507842 2566 patch_prober.go:28] interesting pod/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 06 19:45:45.508274 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:45.507894 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" podUID="e68641b6-f8ad-427e-a7e9-2730513420dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 06 19:45:46.511713 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:46.511685 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:46.512602 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:46.512584 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:48.800584 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:48.800553 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:48.904420 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:48.904390 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:49.112032 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:49.111940 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4"] Jul 06 19:45:49.112214 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:49.112191 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" podUID="e68641b6-f8ad-427e-a7e9-2730513420dd" containerName="istio-proxy" containerID="cri-o://7e01a6cecc43503330fe6f44d190f3d000240299c1c474ed97a13a8623481e0a" gracePeriod=30 Jul 06 19:45:49.153556 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:49.153524 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:49.396403 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:49.396318 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:49.568336 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:49.568299 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:49.844868 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:49.844836 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:49.952892 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:49.952858 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:54.368049 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.368021 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:54.475762 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.475712 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-data\") pod \"e68641b6-f8ad-427e-a7e9-2730513420dd\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " Jul 06 19:45:54.475762 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.475769 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-podinfo\") pod \"e68641b6-f8ad-427e-a7e9-2730513420dd\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " Jul 06 19:45:54.476010 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.475798 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-token\") pod \"e68641b6-f8ad-427e-a7e9-2730513420dd\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " Jul 06 19:45:54.476010 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.475833 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-envoy\") pod \"e68641b6-f8ad-427e-a7e9-2730513420dd\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " Jul 06 19:45:54.476010 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.475859 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68641b6-f8ad-427e-a7e9-2730513420dd-istiod-ca-cert\") pod \"e68641b6-f8ad-427e-a7e9-2730513420dd\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " Jul 06 19:45:54.476010 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.475931 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-workload-certs\") pod \"e68641b6-f8ad-427e-a7e9-2730513420dd\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " Jul 06 19:45:54.476010 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.475978 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-credential-socket\") pod \"e68641b6-f8ad-427e-a7e9-2730513420dd\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " Jul 06 19:45:54.476283 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.476025 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-workload-socket\") pod \"e68641b6-f8ad-427e-a7e9-2730513420dd\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " Jul 06 19:45:54.476283 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.476075 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctpsh\" (UniqueName: \"kubernetes.io/projected/e68641b6-f8ad-427e-a7e9-2730513420dd-kube-api-access-ctpsh\") pod \"e68641b6-f8ad-427e-a7e9-2730513420dd\" (UID: \"e68641b6-f8ad-427e-a7e9-2730513420dd\") " Jul 06 19:45:54.476283 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.476230 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68641b6-f8ad-427e-a7e9-2730513420dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e68641b6-f8ad-427e-a7e9-2730513420dd" (UID: "e68641b6-f8ad-427e-a7e9-2730513420dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:45:54.476432 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.476327 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e68641b6-f8ad-427e-a7e9-2730513420dd" (UID: "e68641b6-f8ad-427e-a7e9-2730513420dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:45:54.476490 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.476428 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68641b6-f8ad-427e-a7e9-2730513420dd-istiod-ca-cert\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:45:54.476490 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.476448 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-workload-socket\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:45:54.476592 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.476533 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "e68641b6-f8ad-427e-a7e9-2730513420dd" (UID: "e68641b6-f8ad-427e-a7e9-2730513420dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:45:54.476691 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.476665 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e68641b6-f8ad-427e-a7e9-2730513420dd" (UID: "e68641b6-f8ad-427e-a7e9-2730513420dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:45:54.476893 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.476867 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e68641b6-f8ad-427e-a7e9-2730513420dd" (UID: "e68641b6-f8ad-427e-a7e9-2730513420dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:45:54.478178 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.478152 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e68641b6-f8ad-427e-a7e9-2730513420dd" (UID: "e68641b6-f8ad-427e-a7e9-2730513420dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:45:54.478178 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.478164 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "e68641b6-f8ad-427e-a7e9-2730513420dd" (UID: "e68641b6-f8ad-427e-a7e9-2730513420dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:45:54.478348 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.478241 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68641b6-f8ad-427e-a7e9-2730513420dd-kube-api-access-ctpsh" (OuterVolumeSpecName: "kube-api-access-ctpsh") pod "e68641b6-f8ad-427e-a7e9-2730513420dd" (UID: "e68641b6-f8ad-427e-a7e9-2730513420dd"). InnerVolumeSpecName "kube-api-access-ctpsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:45:54.478348 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.478263 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e68641b6-f8ad-427e-a7e9-2730513420dd" (UID: "e68641b6-f8ad-427e-a7e9-2730513420dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:45:54.577260 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.577231 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctpsh\" (UniqueName: \"kubernetes.io/projected/e68641b6-f8ad-427e-a7e9-2730513420dd-kube-api-access-ctpsh\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:45:54.577260 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.577260 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-data\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:45:54.577463 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.577272 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-podinfo\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:45:54.577463 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.577283 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-token\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:45:54.577463 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.577298 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-istio-envoy\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:45:54.577463 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.577308 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-workload-certs\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:45:54.577463 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.577318 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68641b6-f8ad-427e-a7e9-2730513420dd-credential-socket\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:45:54.756279 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.756189 2566 generic.go:356] "Generic (PLEG): container finished" podID="e68641b6-f8ad-427e-a7e9-2730513420dd" containerID="7e01a6cecc43503330fe6f44d190f3d000240299c1c474ed97a13a8623481e0a" exitCode=0 Jul 06 19:45:54.756279 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.756236 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" Jul 06 19:45:54.756483 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.756275 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" event={"ID":"e68641b6-f8ad-427e-a7e9-2730513420dd","Type":"ContainerDied","Data":"7e01a6cecc43503330fe6f44d190f3d000240299c1c474ed97a13a8623481e0a"} Jul 06 19:45:54.756483 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.756334 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4" event={"ID":"e68641b6-f8ad-427e-a7e9-2730513420dd","Type":"ContainerDied","Data":"93d458c8cc31dd42a689f60fe5ff5d9878294864e572fed7945364549ff431e5"} Jul 06 19:45:54.756483 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.756357 2566 scope.go:117] "RemoveContainer" containerID="7e01a6cecc43503330fe6f44d190f3d000240299c1c474ed97a13a8623481e0a" Jul 06 19:45:54.765882 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.765863 2566 scope.go:117] "RemoveContainer" containerID="452f0ad83138cbc0c2f1f463bdf3462cc2606603ee26cfc251c9507452378aa4" Jul 06 19:45:54.773816 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.773802 2566 scope.go:117] "RemoveContainer" containerID="7e01a6cecc43503330fe6f44d190f3d000240299c1c474ed97a13a8623481e0a" Jul 06 19:45:54.774071 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:45:54.774055 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e01a6cecc43503330fe6f44d190f3d000240299c1c474ed97a13a8623481e0a\": container with ID starting with 7e01a6cecc43503330fe6f44d190f3d000240299c1c474ed97a13a8623481e0a not found: ID does not exist" containerID="7e01a6cecc43503330fe6f44d190f3d000240299c1c474ed97a13a8623481e0a" Jul 06 19:45:54.774112 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.774080 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e01a6cecc43503330fe6f44d190f3d000240299c1c474ed97a13a8623481e0a"} err="failed to get container status \"7e01a6cecc43503330fe6f44d190f3d000240299c1c474ed97a13a8623481e0a\": rpc error: code = NotFound desc = could not find container \"7e01a6cecc43503330fe6f44d190f3d000240299c1c474ed97a13a8623481e0a\": container with ID starting with 7e01a6cecc43503330fe6f44d190f3d000240299c1c474ed97a13a8623481e0a not found: ID does not exist" Jul 06 19:45:54.774112 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.774097 2566 scope.go:117] "RemoveContainer" containerID="452f0ad83138cbc0c2f1f463bdf3462cc2606603ee26cfc251c9507452378aa4" Jul 06 19:45:54.774329 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:45:54.774309 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"452f0ad83138cbc0c2f1f463bdf3462cc2606603ee26cfc251c9507452378aa4\": container with ID starting with 452f0ad83138cbc0c2f1f463bdf3462cc2606603ee26cfc251c9507452378aa4 not found: ID does not exist" containerID="452f0ad83138cbc0c2f1f463bdf3462cc2606603ee26cfc251c9507452378aa4" Jul 06 19:45:54.774423 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.774331 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452f0ad83138cbc0c2f1f463bdf3462cc2606603ee26cfc251c9507452378aa4"} err="failed to get container status \"452f0ad83138cbc0c2f1f463bdf3462cc2606603ee26cfc251c9507452378aa4\": rpc error: code = NotFound desc = could not find container \"452f0ad83138cbc0c2f1f463bdf3462cc2606603ee26cfc251c9507452378aa4\": container with ID starting with 452f0ad83138cbc0c2f1f463bdf3462cc2606603ee26cfc251c9507452378aa4 not found: ID does not exist" Jul 06 19:45:54.781015 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.780986 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4"] Jul 06 19:45:54.785554 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:54.785535 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b3c8385f-openshift-default-5b7c4d9c99-2lsm4"] Jul 06 19:45:55.764370 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:55.764328 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e68641b6-f8ad-427e-a7e9-2730513420dd" path="/var/lib/kubelet/pods/e68641b6-f8ad-427e-a7e9-2730513420dd/volumes" Jul 06 19:45:56.310013 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:56.309978 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:56.529120 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:56.529087 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:56.685807 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:56.685781 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:56.821044 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:45:56.821012 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:02.999332 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:02.999298 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:03.159461 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:03.159408 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:03.498054 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:03.498019 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:03.784262 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:03.784181 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:10.177357 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.177277 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:10.517172 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.517095 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x"] Jul 06 19:46:10.517470 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.517458 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68641b6-f8ad-427e-a7e9-2730513420dd" containerName="istio-proxy" Jul 06 19:46:10.517517 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.517471 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68641b6-f8ad-427e-a7e9-2730513420dd" containerName="istio-proxy" Jul 06 19:46:10.517517 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.517487 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68641b6-f8ad-427e-a7e9-2730513420dd" containerName="istio-proxy" Jul 06 19:46:10.517517 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.517492 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68641b6-f8ad-427e-a7e9-2730513420dd" containerName="istio-proxy" Jul 06 19:46:10.517611 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.517558 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e68641b6-f8ad-427e-a7e9-2730513420dd" containerName="istio-proxy" Jul 06 19:46:10.517611 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.517565 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e68641b6-f8ad-427e-a7e9-2730513420dd" containerName="istio-proxy" Jul 06 19:46:10.520755 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.520735 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.524909 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.524891 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9349783c-openshift-default-dockercfg-zttsl\"" Jul 06 19:46:10.559191 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.559154 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x"] Jul 06 19:46:10.570228 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.570180 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:10.626045 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.625997 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b18d57f5-6696-4063-b2f1-7da98492e577-istio-token\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.626209 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.626093 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-credential-socket\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.626209 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.626152 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b18d57f5-6696-4063-b2f1-7da98492e577-istiod-ca-cert\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.626209 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.626175 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-istio-envoy\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.626322 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.626209 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc94z\" (UniqueName: \"kubernetes.io/projected/b18d57f5-6696-4063-b2f1-7da98492e577-kube-api-access-rc94z\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.626322 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.626303 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-workload-socket\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.626384 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.626323 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-istio-data\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.626448 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.626426 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-workload-certs\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.626487 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.626464 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b18d57f5-6696-4063-b2f1-7da98492e577-istio-podinfo\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.727111 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.727062 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-credential-socket\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.727309 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.727155 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b18d57f5-6696-4063-b2f1-7da98492e577-istiod-ca-cert\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.727309 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.727187 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-istio-envoy\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.727309 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.727232 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc94z\" (UniqueName: \"kubernetes.io/projected/b18d57f5-6696-4063-b2f1-7da98492e577-kube-api-access-rc94z\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.727309 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.727293 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-workload-socket\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.727530 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.727318 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-istio-data\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.727530 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.727375 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-workload-certs\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.727530 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.727401 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b18d57f5-6696-4063-b2f1-7da98492e577-istio-podinfo\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.727530 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.727430 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b18d57f5-6696-4063-b2f1-7da98492e577-istio-token\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.727848 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.727821 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-credential-socket\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.727848 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.727840 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-workload-socket\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.728038 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.728017 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-istio-data\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.728105 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.728086 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-workload-certs\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.728776 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.728742 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b18d57f5-6696-4063-b2f1-7da98492e577-istiod-ca-cert\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.729988 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.729960 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-istio-envoy\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.730827 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.730805 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b18d57f5-6696-4063-b2f1-7da98492e577-istio-podinfo\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.739138 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.739106 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc94z\" (UniqueName: \"kubernetes.io/projected/b18d57f5-6696-4063-b2f1-7da98492e577-kube-api-access-rc94z\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.741144 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.741115 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b18d57f5-6696-4063-b2f1-7da98492e577-istio-token\") pod \"e2e-mt-9349783c-openshift-default-979fc98c-9mv4x\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:10.835530 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:10.835429 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:11.005258 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:11.005213 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x"] Jul 06 19:46:11.008886 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:46:11.008844 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18d57f5_6696_4063_b2f1_7da98492e577.slice/crio-9ba8992e626efcbef2ac2d6e36229359bd1cbe006e32dbf4ccae0f411efd60f1 WatchSource:0}: Error finding container 9ba8992e626efcbef2ac2d6e36229359bd1cbe006e32dbf4ccae0f411efd60f1: Status 404 returned error can't find the container with id 9ba8992e626efcbef2ac2d6e36229359bd1cbe006e32dbf4ccae0f411efd60f1 Jul 06 19:46:11.011213 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:11.011170 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:46:11.011313 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:11.011278 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:46:11.011355 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:11.011326 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:46:11.043632 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:11.043602 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:11.690810 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:11.690762 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:11.865501 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:11.865458 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" event={"ID":"b18d57f5-6696-4063-b2f1-7da98492e577","Type":"ContainerStarted","Data":"55c2814db5f623967b94dab84d0597478c03fd197eff50ed031204e2110c3767"} Jul 06 19:46:11.865687 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:11.865515 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" event={"ID":"b18d57f5-6696-4063-b2f1-7da98492e577","Type":"ContainerStarted","Data":"9ba8992e626efcbef2ac2d6e36229359bd1cbe006e32dbf4ccae0f411efd60f1"} Jul 06 19:46:12.836310 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:12.836278 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:12.840968 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:12.840943 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:12.864243 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:12.864195 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" podStartSLOduration=2.8641801620000003 podStartE2EDuration="2.864180162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:46:11.900717183 +0000 UTC m=+2068.796606717" watchObservedRunningTime="2026-07-06 19:46:12.864180162 +0000 UTC m=+2069.760069771" Jul 06 19:46:12.870974 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:12.870943 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:12.871989 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:12.871971 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:13.905453 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:13.905423 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:14.044570 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:14.044531 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:14.137976 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:14.137937 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:14.255410 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:14.255378 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:14.828113 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:14.828076 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:14.924190 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:14.924157 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:18.446062 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:18.446023 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:24.973919 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:24.973824 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:27.872617 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:27.872522 2566 patch_prober.go:28] interesting pod/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 06 19:46:27.872617 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:27.872576 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" podUID="b18d57f5-6696-4063-b2f1-7da98492e577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 06 19:46:27.962849 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:27.962819 2566 generic.go:356] "Generic (PLEG): container finished" podID="b18d57f5-6696-4063-b2f1-7da98492e577" containerID="55c2814db5f623967b94dab84d0597478c03fd197eff50ed031204e2110c3767" exitCode=0 Jul 06 19:46:27.963020 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:27.962894 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" event={"ID":"b18d57f5-6696-4063-b2f1-7da98492e577","Type":"ContainerDied","Data":"55c2814db5f623967b94dab84d0597478c03fd197eff50ed031204e2110c3767"} Jul 06 19:46:27.963371 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:27.963354 2566 scope.go:117] "RemoveContainer" containerID="55c2814db5f623967b94dab84d0597478c03fd197eff50ed031204e2110c3767" Jul 06 19:46:27.964382 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:27.964355 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:46:27.964434 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:27.964423 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:46:27.964482 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:27.964458 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:46:28.970259 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:28.970223 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" event={"ID":"b18d57f5-6696-4063-b2f1-7da98492e577","Type":"ContainerStarted","Data":"fc78b1f8d069a994600a9992f3dc5b1a7080e812f345ef8db869da40992b6542"} Jul 06 19:46:29.836513 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:29.836471 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:29.836761 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:29.836597 2566 patch_prober.go:28] interesting pod/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 06 19:46:29.836761 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:29.836641 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" podUID="b18d57f5-6696-4063-b2f1-7da98492e577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 06 19:46:30.836415 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:30.836369 2566 patch_prober.go:28] interesting pod/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 06 19:46:30.836928 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:30.836438 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" podUID="b18d57f5-6696-4063-b2f1-7da98492e577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 06 19:46:31.836832 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:31.836790 2566 patch_prober.go:28] interesting pod/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 06 19:46:31.837333 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:31.836854 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" podUID="b18d57f5-6696-4063-b2f1-7da98492e577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 06 19:46:32.839456 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:32.839427 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:32.839868 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:32.839694 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:32.840375 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:32.840360 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:46:33.612897 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:33.612863 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:43.765290 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:43.765261 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:46:43.795440 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:43.795416 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:46:45.994995 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:45.994950 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:46.588337 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:46.588301 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:52.138694 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:52.138657 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:57.289660 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:46:57.289615 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:47:34.262201 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:47:34.262165 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:47:35.565435 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:47:35.565338 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:47:41.659194 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:47:41.659152 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:47:57.216397 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:47:57.216358 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:47:57.339330 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:47:57.339290 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:47:57.574923 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:47:57.574816 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x"] Jul 06 19:47:57.575135 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:47:57.575102 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" podUID="b18d57f5-6696-4063-b2f1-7da98492e577" containerName="istio-proxy" containerID="cri-o://fc78b1f8d069a994600a9992f3dc5b1a7080e812f345ef8db869da40992b6542" gracePeriod=30 Jul 06 19:47:57.601085 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:47:57.601053 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:47:57.879584 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:47:57.879427 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:47:59.095131 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:47:59.095095 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:47:59.271260 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:47:59.271226 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:48:00.161368 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:00.161340 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:48:00.364422 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:00.364383 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:48:01.848535 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:01.848501 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:48:02.844229 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.844161 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:48:02.867053 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.867013 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b18d57f5-6696-4063-b2f1-7da98492e577-istio-podinfo\") pod \"b18d57f5-6696-4063-b2f1-7da98492e577\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " Jul 06 19:48:02.867053 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.867051 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b18d57f5-6696-4063-b2f1-7da98492e577-istio-token\") pod \"b18d57f5-6696-4063-b2f1-7da98492e577\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " Jul 06 19:48:02.867847 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.867085 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc94z\" (UniqueName: \"kubernetes.io/projected/b18d57f5-6696-4063-b2f1-7da98492e577-kube-api-access-rc94z\") pod \"b18d57f5-6696-4063-b2f1-7da98492e577\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " Jul 06 19:48:02.867847 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.867127 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b18d57f5-6696-4063-b2f1-7da98492e577-istiod-ca-cert\") pod \"b18d57f5-6696-4063-b2f1-7da98492e577\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " Jul 06 19:48:02.867847 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.867151 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-istio-data\") pod \"b18d57f5-6696-4063-b2f1-7da98492e577\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " Jul 06 19:48:02.867847 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.867180 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-istio-envoy\") pod \"b18d57f5-6696-4063-b2f1-7da98492e577\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " Jul 06 19:48:02.867847 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.867226 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-workload-certs\") pod \"b18d57f5-6696-4063-b2f1-7da98492e577\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " Jul 06 19:48:02.867847 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.867258 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-credential-socket\") pod \"b18d57f5-6696-4063-b2f1-7da98492e577\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " Jul 06 19:48:02.867847 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.867322 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-workload-socket\") pod \"b18d57f5-6696-4063-b2f1-7da98492e577\" (UID: \"b18d57f5-6696-4063-b2f1-7da98492e577\") " Jul 06 19:48:02.868241 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.867939 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b18d57f5-6696-4063-b2f1-7da98492e577" (UID: "b18d57f5-6696-4063-b2f1-7da98492e577"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:48:02.868241 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.868127 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b18d57f5-6696-4063-b2f1-7da98492e577" (UID: "b18d57f5-6696-4063-b2f1-7da98492e577"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:48:02.870296 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.870202 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-istio-data" (OuterVolumeSpecName: "istio-data") pod "b18d57f5-6696-4063-b2f1-7da98492e577" (UID: "b18d57f5-6696-4063-b2f1-7da98492e577"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:48:02.870764 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.870702 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b18d57f5-6696-4063-b2f1-7da98492e577-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b18d57f5-6696-4063-b2f1-7da98492e577" (UID: "b18d57f5-6696-4063-b2f1-7da98492e577"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:48:02.871276 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.871232 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b18d57f5-6696-4063-b2f1-7da98492e577-kube-api-access-rc94z" (OuterVolumeSpecName: "kube-api-access-rc94z") pod "b18d57f5-6696-4063-b2f1-7da98492e577" (UID: "b18d57f5-6696-4063-b2f1-7da98492e577"). InnerVolumeSpecName "kube-api-access-rc94z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:48:02.871276 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.871234 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b18d57f5-6696-4063-b2f1-7da98492e577" (UID: "b18d57f5-6696-4063-b2f1-7da98492e577"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:48:02.871540 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.871510 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b18d57f5-6696-4063-b2f1-7da98492e577-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b18d57f5-6696-4063-b2f1-7da98492e577" (UID: "b18d57f5-6696-4063-b2f1-7da98492e577"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:48:02.871858 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.871839 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b18d57f5-6696-4063-b2f1-7da98492e577-istio-token" (OuterVolumeSpecName: "istio-token") pod "b18d57f5-6696-4063-b2f1-7da98492e577" (UID: "b18d57f5-6696-4063-b2f1-7da98492e577"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:48:02.873504 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.873479 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b18d57f5-6696-4063-b2f1-7da98492e577" (UID: "b18d57f5-6696-4063-b2f1-7da98492e577"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:48:02.968810 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.968695 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc94z\" (UniqueName: \"kubernetes.io/projected/b18d57f5-6696-4063-b2f1-7da98492e577-kube-api-access-rc94z\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:48:02.968810 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.968739 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b18d57f5-6696-4063-b2f1-7da98492e577-istiod-ca-cert\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:48:02.968810 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.968754 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-istio-data\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:48:02.968810 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.968765 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-istio-envoy\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:48:02.968810 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.968783 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-workload-certs\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:48:02.968810 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.968791 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-credential-socket\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:48:02.968810 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.968799 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b18d57f5-6696-4063-b2f1-7da98492e577-workload-socket\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:48:02.968810 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.968808 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b18d57f5-6696-4063-b2f1-7da98492e577-istio-podinfo\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:48:02.968810 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:02.968816 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b18d57f5-6696-4063-b2f1-7da98492e577-istio-token\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:48:03.480074 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:03.480042 2566 generic.go:356] "Generic (PLEG): container finished" podID="b18d57f5-6696-4063-b2f1-7da98492e577" containerID="fc78b1f8d069a994600a9992f3dc5b1a7080e812f345ef8db869da40992b6542" exitCode=0 Jul 06 19:48:03.480290 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:03.480094 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" Jul 06 19:48:03.480290 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:03.480125 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" event={"ID":"b18d57f5-6696-4063-b2f1-7da98492e577","Type":"ContainerDied","Data":"fc78b1f8d069a994600a9992f3dc5b1a7080e812f345ef8db869da40992b6542"} Jul 06 19:48:03.480290 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:03.480176 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" event={"ID":"b18d57f5-6696-4063-b2f1-7da98492e577","Type":"ContainerDied","Data":"9ba8992e626efcbef2ac2d6e36229359bd1cbe006e32dbf4ccae0f411efd60f1"} Jul 06 19:48:03.480290 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:03.480199 2566 scope.go:117] "RemoveContainer" containerID="fc78b1f8d069a994600a9992f3dc5b1a7080e812f345ef8db869da40992b6542" Jul 06 19:48:03.489961 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:03.489940 2566 scope.go:117] "RemoveContainer" containerID="55c2814db5f623967b94dab84d0597478c03fd197eff50ed031204e2110c3767" Jul 06 19:48:03.498177 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:03.498159 2566 scope.go:117] "RemoveContainer" containerID="fc78b1f8d069a994600a9992f3dc5b1a7080e812f345ef8db869da40992b6542" Jul 06 19:48:03.498440 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:48:03.498421 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc78b1f8d069a994600a9992f3dc5b1a7080e812f345ef8db869da40992b6542\": container with ID starting with fc78b1f8d069a994600a9992f3dc5b1a7080e812f345ef8db869da40992b6542 not found: ID does not exist" containerID="fc78b1f8d069a994600a9992f3dc5b1a7080e812f345ef8db869da40992b6542" Jul 06 19:48:03.498488 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:03.498449 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc78b1f8d069a994600a9992f3dc5b1a7080e812f345ef8db869da40992b6542"} err="failed to get container status \"fc78b1f8d069a994600a9992f3dc5b1a7080e812f345ef8db869da40992b6542\": rpc error: code = NotFound desc = could not find container \"fc78b1f8d069a994600a9992f3dc5b1a7080e812f345ef8db869da40992b6542\": container with ID starting with fc78b1f8d069a994600a9992f3dc5b1a7080e812f345ef8db869da40992b6542 not found: ID does not exist" Jul 06 19:48:03.498488 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:03.498467 2566 scope.go:117] "RemoveContainer" containerID="55c2814db5f623967b94dab84d0597478c03fd197eff50ed031204e2110c3767" Jul 06 19:48:03.498704 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:48:03.498686 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55c2814db5f623967b94dab84d0597478c03fd197eff50ed031204e2110c3767\": container with ID starting with 55c2814db5f623967b94dab84d0597478c03fd197eff50ed031204e2110c3767 not found: ID does not exist" containerID="55c2814db5f623967b94dab84d0597478c03fd197eff50ed031204e2110c3767" Jul 06 19:48:03.498765 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:03.498709 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55c2814db5f623967b94dab84d0597478c03fd197eff50ed031204e2110c3767"} err="failed to get container status \"55c2814db5f623967b94dab84d0597478c03fd197eff50ed031204e2110c3767\": rpc error: code = NotFound desc = could not find container \"55c2814db5f623967b94dab84d0597478c03fd197eff50ed031204e2110c3767\": container with ID starting with 55c2814db5f623967b94dab84d0597478c03fd197eff50ed031204e2110c3767 not found: ID does not exist" Jul 06 19:48:03.502912 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:03.502890 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x"] Jul 06 19:48:03.508715 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:03.508694 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x"] Jul 06 19:48:03.764684 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:03.764582 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b18d57f5-6696-4063-b2f1-7da98492e577" path="/var/lib/kubelet/pods/b18d57f5-6696-4063-b2f1-7da98492e577/volumes" Jul 06 19:48:03.840555 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:03.840511 2566 patch_prober.go:28] interesting pod/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 19:48:03.840763 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:03.840593 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9349783c-openshift-default-979fc98c-9mv4x" podUID="b18d57f5-6696-4063-b2f1-7da98492e577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 19:48:08.796301 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:08.796263 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:48:08.897600 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:08.897567 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:48:09.604957 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:09.604920 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:48:16.177168 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:16.177116 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:48:16.328468 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:16.328430 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:48:16.430606 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:16.430509 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:48:26.981492 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:26.981457 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:48:27.289883 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:27.289796 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:48:48.347605 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:48.347566 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:48:55.231072 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:55.231039 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:48:55.416156 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:55.416115 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:48:55.728446 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:55.728402 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:48:56.253546 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.253510 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h"] Jul 06 19:48:56.253984 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.253968 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b18d57f5-6696-4063-b2f1-7da98492e577" containerName="istio-proxy" Jul 06 19:48:56.253984 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.253984 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b18d57f5-6696-4063-b2f1-7da98492e577" containerName="istio-proxy" Jul 06 19:48:56.254103 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.254000 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b18d57f5-6696-4063-b2f1-7da98492e577" containerName="istio-proxy" Jul 06 19:48:56.254103 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.254006 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b18d57f5-6696-4063-b2f1-7da98492e577" containerName="istio-proxy" Jul 06 19:48:56.254103 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.254059 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b18d57f5-6696-4063-b2f1-7da98492e577" containerName="istio-proxy" Jul 06 19:48:56.254103 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.254070 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b18d57f5-6696-4063-b2f1-7da98492e577" containerName="istio-proxy" Jul 06 19:48:56.257170 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.257151 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.263044 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.263015 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d98408-openshift-default-dockercfg-2jfnk\"" Jul 06 19:48:56.285958 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.285928 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:48:56.298173 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.298147 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h"] Jul 06 19:48:56.364900 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.364855 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-envoy\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.364900 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.364911 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj5fs\" (UniqueName: \"kubernetes.io/projected/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-kube-api-access-gj5fs\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.365183 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.364978 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-credential-socket\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.365183 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.365071 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istiod-ca-cert\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.365183 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.365117 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-token\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.365183 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.365159 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-workload-certs\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.365183 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.365182 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-data\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.365414 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.365232 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-workload-socket\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.365414 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.365267 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-podinfo\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.461200 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.461163 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:48:56.466369 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.466343 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj5fs\" (UniqueName: \"kubernetes.io/projected/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-kube-api-access-gj5fs\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.466492 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.466384 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-credential-socket\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.466576 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.466558 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istiod-ca-cert\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.466613 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.466597 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-token\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.466652 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.466633 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-workload-certs\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.466686 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.466655 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-data\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.466737 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.466704 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-workload-socket\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.466789 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.466761 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-podinfo\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.466833 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.466800 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-envoy\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.466833 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.466811 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-credential-socket\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.467106 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.467083 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-data\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.467186 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.467118 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-workload-certs\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.467290 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.467269 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-workload-socket\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.467388 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.467305 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istiod-ca-cert\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.469025 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.469006 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-envoy\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.469231 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.469214 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-podinfo\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.484355 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.484323 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-token\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.486208 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.486183 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj5fs\" (UniqueName: \"kubernetes.io/projected/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-kube-api-access-gj5fs\") pod \"e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.567797 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.567673 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:56.724336 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.724296 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h"] Jul 06 19:48:56.729003 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:48:56.728973 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2a33e7_f42f_4a34_8a59_64317fcc4f5b.slice/crio-527650fa438ff1f444d87538079afe8ccfac41c1da5a6f3cb1d4b1242f31404b WatchSource:0}: Error finding container 527650fa438ff1f444d87538079afe8ccfac41c1da5a6f3cb1d4b1242f31404b: Status 404 returned error can't find the container with id 527650fa438ff1f444d87538079afe8ccfac41c1da5a6f3cb1d4b1242f31404b Jul 06 19:48:56.731358 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.731317 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:48:56.731444 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.731410 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:48:56.731483 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.731458 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:48:56.765628 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:56.765591 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" event={"ID":"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b","Type":"ContainerStarted","Data":"527650fa438ff1f444d87538079afe8ccfac41c1da5a6f3cb1d4b1242f31404b"} Jul 06 19:48:57.772337 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:57.772302 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" event={"ID":"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b","Type":"ContainerStarted","Data":"e19be932687f1967ce9b76704360a70cd96e9bfcf73509810055600d8e623526"} Jul 06 19:48:57.821755 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:57.821689 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" podStartSLOduration=1.821671931 podStartE2EDuration="1.821671931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:48:57.819184485 +0000 UTC m=+2234.715074016" watchObservedRunningTime="2026-07-06 19:48:57.821671931 +0000 UTC m=+2234.717561462" Jul 06 19:48:58.568570 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:58.568534 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:58.573794 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:58.573770 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:58.777639 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:58.777609 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:48:58.778484 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:48:58.778466 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:49:00.117686 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:00.117645 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:00.246935 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:00.246903 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:00.535444 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:00.535414 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:00.783919 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:00.783882 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:01.373471 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:01.373438 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:01.638352 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:01.638267 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:04.567317 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:04.567283 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:04.931495 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:04.931407 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:05.277792 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.277675 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm"] Jul 06 19:49:05.281433 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.281410 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.284540 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.284517 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f1ee26-openshift-default-dockercfg-b98bd\"" Jul 06 19:49:05.310254 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.310210 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm"] Jul 06 19:49:05.327967 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.327929 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:05.349245 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.349210 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-istio-envoy\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.349245 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.349244 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b25862e-3082-4019-abb3-724632fface5-istio-podinfo\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.349503 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.349269 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-workload-certs\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.349503 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.349387 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-credential-socket\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.349503 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.349415 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-istio-data\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.349503 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.349446 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b25862e-3082-4019-abb3-724632fface5-istio-token\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.349503 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.349472 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b25862e-3082-4019-abb3-724632fface5-istiod-ca-cert\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.349773 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.349622 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkfg6\" (UniqueName: \"kubernetes.io/projected/7b25862e-3082-4019-abb3-724632fface5-kube-api-access-rkfg6\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.349773 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.349660 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-workload-socket\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.450523 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.450488 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-istio-envoy\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.450523 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.450525 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b25862e-3082-4019-abb3-724632fface5-istio-podinfo\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.450844 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.450548 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-workload-certs\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.450844 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.450694 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-credential-socket\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.450844 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.450769 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-istio-data\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.450844 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.450815 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b25862e-3082-4019-abb3-724632fface5-istio-token\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.451066 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.450861 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b25862e-3082-4019-abb3-724632fface5-istiod-ca-cert\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.451066 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.450942 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkfg6\" (UniqueName: \"kubernetes.io/projected/7b25862e-3082-4019-abb3-724632fface5-kube-api-access-rkfg6\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.451066 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.450968 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-workload-socket\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.451066 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.451010 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-workload-certs\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.451239 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.451096 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-credential-socket\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.451500 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.451472 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-istio-data\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.451583 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.451511 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-workload-socket\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.451714 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.451691 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b25862e-3082-4019-abb3-724632fface5-istiod-ca-cert\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.453081 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.453059 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-istio-envoy\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.453242 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.453227 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b25862e-3082-4019-abb3-724632fface5-istio-podinfo\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.463157 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.463132 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkfg6\" (UniqueName: \"kubernetes.io/projected/7b25862e-3082-4019-abb3-724632fface5-kube-api-access-rkfg6\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.463340 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.463326 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b25862e-3082-4019-abb3-724632fface5-istio-token\") pod \"e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.593081 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.592983 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:05.632445 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.632407 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:05.776510 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.776352 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm"] Jul 06 19:49:05.777417 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.777379 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:49:05.777502 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.777448 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:49:05.777502 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.777476 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:49:05.821495 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:05.821468 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" event={"ID":"7b25862e-3082-4019-abb3-724632fface5","Type":"ContainerStarted","Data":"218014f9af2fab379c58a2fd3231c31ffd04cfc4b08173ced9077224ad378aae"} Jul 06 19:49:06.831113 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:06.831067 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" event={"ID":"7b25862e-3082-4019-abb3-724632fface5","Type":"ContainerStarted","Data":"5e51a134504a095a80da5435aaa4cb007359a76683089e1efeb49d4f64080c2b"} Jul 06 19:49:06.861982 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:06.861929 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" podStartSLOduration=1.861906513 podStartE2EDuration="1.861906513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:49:06.86027128 +0000 UTC m=+2243.756160825" watchObservedRunningTime="2026-07-06 19:49:06.861906513 +0000 UTC m=+2243.757796045" Jul 06 19:49:07.593426 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:07.593382 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:07.598633 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:07.598605 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:07.840523 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:07.840485 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:07.850302 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:07.850209 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:09.043971 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:09.043938 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:09.229915 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:09.229879 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:09.532515 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:09.532482 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:10.222440 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:10.222412 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:10.369295 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:10.369195 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:13.440500 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:13.440467 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:13.778987 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:13.778898 2566 patch_prober.go:28] interesting pod/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 06 19:49:13.778987 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:13.778958 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" podUID="bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 06 19:49:13.881951 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:13.881918 2566 generic.go:356] "Generic (PLEG): container finished" podID="bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" containerID="e19be932687f1967ce9b76704360a70cd96e9bfcf73509810055600d8e623526" exitCode=0 Jul 06 19:49:13.882123 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:13.881994 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" event={"ID":"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b","Type":"ContainerDied","Data":"e19be932687f1967ce9b76704360a70cd96e9bfcf73509810055600d8e623526"} Jul 06 19:49:13.882432 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:13.882412 2566 scope.go:117] "RemoveContainer" containerID="e19be932687f1967ce9b76704360a70cd96e9bfcf73509810055600d8e623526" Jul 06 19:49:13.883554 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:13.883523 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:49:13.883630 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:13.883594 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:49:13.883630 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:13.883626 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:49:14.889068 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:14.889030 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" event={"ID":"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b","Type":"ContainerStarted","Data":"3178319d9499e46c2b8a78928ec08b0a1f2d0839936836dec7a18c9ee00df0ce"} Jul 06 19:49:15.568571 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:15.568534 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:49:15.568833 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:15.568807 2566 patch_prober.go:28] interesting pod/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 06 19:49:15.568945 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:15.568862 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" podUID="bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 06 19:49:16.569204 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:16.569159 2566 patch_prober.go:28] interesting pod/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 06 19:49:16.569808 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:16.569229 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" podUID="bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 06 19:49:17.568495 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:17.568453 2566 patch_prober.go:28] interesting pod/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 06 19:49:17.568837 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:17.568804 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" podUID="bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 06 19:49:18.568507 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:18.568464 2566 patch_prober.go:28] interesting pod/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 06 19:49:18.569145 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:18.569109 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" podUID="bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 06 19:49:19.572639 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:19.572603 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:49:19.573113 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:19.573014 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:49:19.573631 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:19.573614 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:49:23.417737 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:23.416626 2566 patch_prober.go:28] interesting pod/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": read tcp 10.134.0.2:55480->10.134.0.59:15021: read: connection reset by peer" start-of-body= Jul 06 19:49:23.417737 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:23.416686 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" podUID="7b25862e-3082-4019-abb3-724632fface5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": read tcp 10.134.0.2:55480->10.134.0.59:15021: read: connection reset by peer" Jul 06 19:49:23.849898 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:23.849808 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:23.942500 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:23.942464 2566 generic.go:356] "Generic (PLEG): container finished" podID="7b25862e-3082-4019-abb3-724632fface5" containerID="5e51a134504a095a80da5435aaa4cb007359a76683089e1efeb49d4f64080c2b" exitCode=0 Jul 06 19:49:23.942676 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:23.942538 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" event={"ID":"7b25862e-3082-4019-abb3-724632fface5","Type":"ContainerDied","Data":"5e51a134504a095a80da5435aaa4cb007359a76683089e1efeb49d4f64080c2b"} Jul 06 19:49:23.942957 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:23.942943 2566 scope.go:117] "RemoveContainer" containerID="5e51a134504a095a80da5435aaa4cb007359a76683089e1efeb49d4f64080c2b" Jul 06 19:49:23.944070 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:23.944041 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:49:23.944137 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:23.944107 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:49:23.944174 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:23.944138 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:49:23.945488 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:23.945465 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:24.951001 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:24.950966 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" event={"ID":"7b25862e-3082-4019-abb3-724632fface5","Type":"ContainerStarted","Data":"407169fb7e60c5d51c5bab2b4ad7873aa68288bedec77eef80984406b9a77749"} Jul 06 19:49:25.532667 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:25.532633 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:25.593757 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:25.593689 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:25.593947 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:25.593912 2566 patch_prober.go:28] interesting pod/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 06 19:49:25.594000 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:25.593962 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" podUID="7b25862e-3082-4019-abb3-724632fface5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 06 19:49:26.593700 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:26.593660 2566 patch_prober.go:28] interesting pod/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 06 19:49:26.594206 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:26.593752 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" podUID="7b25862e-3082-4019-abb3-724632fface5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 06 19:49:27.593709 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:27.593672 2566 patch_prober.go:28] interesting pod/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 06 19:49:27.594202 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:27.593762 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" podUID="7b25862e-3082-4019-abb3-724632fface5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 06 19:49:28.593926 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:28.593885 2566 patch_prober.go:28] interesting pod/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 06 19:49:28.594334 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:28.593952 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" podUID="7b25862e-3082-4019-abb3-724632fface5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 06 19:49:29.596984 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:29.596945 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:29.597439 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:29.597411 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:29.598055 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:29.598035 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:49:31.095075 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:31.095031 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:31.190940 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:31.190911 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:40.616486 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:40.616450 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:50.492316 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:49:50.492281 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:12.254022 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:50:12.253988 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:13.224865 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:50:13.224828 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:13.383829 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:50:13.383799 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:13.776272 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:50:13.776236 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:13.876539 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:50:13.876508 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:13.977000 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:50:13.976961 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:14.095036 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:50:14.094945 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:18.848147 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:50:18.848114 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:56.061909 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:50:56.061834 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:56.324284 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:50:56.324207 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h"] Jul 06 19:50:56.324441 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:50:56.324403 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" podUID="bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" containerName="istio-proxy" containerID="cri-o://3178319d9499e46c2b8a78928ec08b0a1f2d0839936836dec7a18c9ee00df0ce" gracePeriod=30 Jul 06 19:50:56.344461 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:50:56.344426 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:56.611758 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:50:56.611661 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:56.809544 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:50:56.809515 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:57.248136 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:50:57.248105 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:57.349043 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:50:57.349008 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:57.531660 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:50:57.531567 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:01.491473 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.491439 2566 generic.go:356] "Generic (PLEG): container finished" podID="bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" containerID="3178319d9499e46c2b8a78928ec08b0a1f2d0839936836dec7a18c9ee00df0ce" exitCode=0 Jul 06 19:51:01.491915 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.491521 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" event={"ID":"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b","Type":"ContainerDied","Data":"3178319d9499e46c2b8a78928ec08b0a1f2d0839936836dec7a18c9ee00df0ce"} Jul 06 19:51:01.491915 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.491605 2566 scope.go:117] "RemoveContainer" containerID="e19be932687f1967ce9b76704360a70cd96e9bfcf73509810055600d8e623526" Jul 06 19:51:01.587587 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.587527 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:51:01.677461 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.677426 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-data\") pod \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " Jul 06 19:51:01.677461 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.677465 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-podinfo\") pod \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " Jul 06 19:51:01.677681 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.677485 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-token\") pod \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " Jul 06 19:51:01.677681 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.677500 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-workload-certs\") pod \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " Jul 06 19:51:01.677681 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.677546 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istiod-ca-cert\") pod \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " Jul 06 19:51:01.677681 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.677598 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj5fs\" (UniqueName: \"kubernetes.io/projected/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-kube-api-access-gj5fs\") pod \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " Jul 06 19:51:01.677681 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.677649 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-credential-socket\") pod \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " Jul 06 19:51:01.677996 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.677688 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-workload-socket\") pod \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " Jul 06 19:51:01.677996 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.677744 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-envoy\") pod \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\" (UID: \"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b\") " Jul 06 19:51:01.677996 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.677882 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" (UID: "bf2a33e7-f42f-4a34-8a59-64317fcc4f5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:01.678162 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.678005 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" (UID: "bf2a33e7-f42f-4a34-8a59-64317fcc4f5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:51:01.678162 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.678111 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" (UID: "bf2a33e7-f42f-4a34-8a59-64317fcc4f5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:01.678274 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.678195 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-workload-certs\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:01.678274 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.678207 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" (UID: "bf2a33e7-f42f-4a34-8a59-64317fcc4f5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:01.678274 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.678214 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istiod-ca-cert\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:01.678274 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.678240 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-workload-socket\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:01.678483 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.678338 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-data" (OuterVolumeSpecName: "istio-data") pod "bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" (UID: "bf2a33e7-f42f-4a34-8a59-64317fcc4f5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:01.679839 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.679808 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-token" (OuterVolumeSpecName: "istio-token") pod "bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" (UID: "bf2a33e7-f42f-4a34-8a59-64317fcc4f5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:51:01.679973 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.679849 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-kube-api-access-gj5fs" (OuterVolumeSpecName: "kube-api-access-gj5fs") pod "bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" (UID: "bf2a33e7-f42f-4a34-8a59-64317fcc4f5b"). InnerVolumeSpecName "kube-api-access-gj5fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:51:01.680196 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.680176 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" (UID: "bf2a33e7-f42f-4a34-8a59-64317fcc4f5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:01.680254 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.680206 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" (UID: "bf2a33e7-f42f-4a34-8a59-64317fcc4f5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:51:01.779451 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.779427 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-token\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:01.779451 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.779452 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj5fs\" (UniqueName: \"kubernetes.io/projected/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-kube-api-access-gj5fs\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:01.779605 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.779462 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-credential-socket\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:01.779605 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.779471 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-envoy\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:01.779605 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.779479 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-data\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:01.779605 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:01.779486 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b-istio-podinfo\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:02.500137 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:02.500105 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" Jul 06 19:51:02.500562 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:02.500148 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h" event={"ID":"bf2a33e7-f42f-4a34-8a59-64317fcc4f5b","Type":"ContainerDied","Data":"527650fa438ff1f444d87538079afe8ccfac41c1da5a6f3cb1d4b1242f31404b"} Jul 06 19:51:02.500562 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:02.500203 2566 scope.go:117] "RemoveContainer" containerID="3178319d9499e46c2b8a78928ec08b0a1f2d0839936836dec7a18c9ee00df0ce" Jul 06 19:51:02.520077 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:02.520050 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h"] Jul 06 19:51:02.524591 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:02.524557 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d98408-openshift-default-fbc9dfc55-mct4h"] Jul 06 19:51:03.392672 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:03.392627 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:03.475186 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:03.475152 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:03.670634 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:03.670601 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm"] Jul 06 19:51:03.671107 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:03.670839 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" podUID="7b25862e-3082-4019-abb3-724632fface5" containerName="istio-proxy" containerID="cri-o://407169fb7e60c5d51c5bab2b4ad7873aa68288bedec77eef80984406b9a77749" gracePeriod=30 Jul 06 19:51:03.765218 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:03.765182 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" path="/var/lib/kubelet/pods/bf2a33e7-f42f-4a34-8a59-64317fcc4f5b/volumes" Jul 06 19:51:03.999432 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:03.999332 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:04.102612 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:04.102579 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:04.359360 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:04.359268 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:04.709572 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:04.709531 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:04.854737 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:04.854701 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:04.961357 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:04.961265 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:05.056713 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:05.056680 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:08.928436 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:08.928395 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:51:09.045226 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.045133 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-credential-socket\") pod \"7b25862e-3082-4019-abb3-724632fface5\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " Jul 06 19:51:09.045226 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.045170 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkfg6\" (UniqueName: \"kubernetes.io/projected/7b25862e-3082-4019-abb3-724632fface5-kube-api-access-rkfg6\") pod \"7b25862e-3082-4019-abb3-724632fface5\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " Jul 06 19:51:09.045226 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.045198 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-workload-certs\") pod \"7b25862e-3082-4019-abb3-724632fface5\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " Jul 06 19:51:09.045512 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.045238 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b25862e-3082-4019-abb3-724632fface5-istiod-ca-cert\") pod \"7b25862e-3082-4019-abb3-724632fface5\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " Jul 06 19:51:09.045512 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.045271 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-workload-socket\") pod \"7b25862e-3082-4019-abb3-724632fface5\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " Jul 06 19:51:09.045512 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.045297 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-istio-envoy\") pod \"7b25862e-3082-4019-abb3-724632fface5\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " Jul 06 19:51:09.045512 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.045352 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b25862e-3082-4019-abb3-724632fface5-istio-podinfo\") pod \"7b25862e-3082-4019-abb3-724632fface5\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " Jul 06 19:51:09.045512 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.045374 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-istio-data\") pod \"7b25862e-3082-4019-abb3-724632fface5\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " Jul 06 19:51:09.045512 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.045460 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b25862e-3082-4019-abb3-724632fface5-istio-token\") pod \"7b25862e-3082-4019-abb3-724632fface5\" (UID: \"7b25862e-3082-4019-abb3-724632fface5\") " Jul 06 19:51:09.045512 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.045456 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7b25862e-3082-4019-abb3-724632fface5" (UID: "7b25862e-3082-4019-abb3-724632fface5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:09.045921 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.045602 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7b25862e-3082-4019-abb3-724632fface5" (UID: "7b25862e-3082-4019-abb3-724632fface5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:09.045921 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.045672 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7b25862e-3082-4019-abb3-724632fface5" (UID: "7b25862e-3082-4019-abb3-724632fface5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:09.045921 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.045703 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b25862e-3082-4019-abb3-724632fface5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7b25862e-3082-4019-abb3-724632fface5" (UID: "7b25862e-3082-4019-abb3-724632fface5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:51:09.045921 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.045849 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-credential-socket\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:09.045921 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.045872 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-workload-certs\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:09.045921 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.045887 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b25862e-3082-4019-abb3-724632fface5-istiod-ca-cert\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:09.045921 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.045899 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-workload-socket\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:09.046369 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.046345 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-istio-data" (OuterVolumeSpecName: "istio-data") pod "7b25862e-3082-4019-abb3-724632fface5" (UID: "7b25862e-3082-4019-abb3-724632fface5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:09.047650 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.047627 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7b25862e-3082-4019-abb3-724632fface5" (UID: "7b25862e-3082-4019-abb3-724632fface5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:09.047766 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.047687 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b25862e-3082-4019-abb3-724632fface5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7b25862e-3082-4019-abb3-724632fface5" (UID: "7b25862e-3082-4019-abb3-724632fface5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:51:09.047990 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.047965 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b25862e-3082-4019-abb3-724632fface5-istio-token" (OuterVolumeSpecName: "istio-token") pod "7b25862e-3082-4019-abb3-724632fface5" (UID: "7b25862e-3082-4019-abb3-724632fface5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:51:09.048087 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.048010 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b25862e-3082-4019-abb3-724632fface5-kube-api-access-rkfg6" (OuterVolumeSpecName: "kube-api-access-rkfg6") pod "7b25862e-3082-4019-abb3-724632fface5" (UID: "7b25862e-3082-4019-abb3-724632fface5"). InnerVolumeSpecName "kube-api-access-rkfg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:51:09.146599 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.146561 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-istio-envoy\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:09.146599 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.146592 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b25862e-3082-4019-abb3-724632fface5-istio-podinfo\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:09.146599 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.146601 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b25862e-3082-4019-abb3-724632fface5-istio-data\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:09.146599 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.146609 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b25862e-3082-4019-abb3-724632fface5-istio-token\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:09.146934 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.146623 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkfg6\" (UniqueName: \"kubernetes.io/projected/7b25862e-3082-4019-abb3-724632fface5-kube-api-access-rkfg6\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:09.541161 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.541125 2566 generic.go:356] "Generic (PLEG): container finished" podID="7b25862e-3082-4019-abb3-724632fface5" containerID="407169fb7e60c5d51c5bab2b4ad7873aa68288bedec77eef80984406b9a77749" exitCode=0 Jul 06 19:51:09.541364 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.541175 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" event={"ID":"7b25862e-3082-4019-abb3-724632fface5","Type":"ContainerDied","Data":"407169fb7e60c5d51c5bab2b4ad7873aa68288bedec77eef80984406b9a77749"} Jul 06 19:51:09.541364 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.541183 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" Jul 06 19:51:09.541364 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.541218 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm" event={"ID":"7b25862e-3082-4019-abb3-724632fface5","Type":"ContainerDied","Data":"218014f9af2fab379c58a2fd3231c31ffd04cfc4b08173ced9077224ad378aae"} Jul 06 19:51:09.541364 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.541240 2566 scope.go:117] "RemoveContainer" containerID="407169fb7e60c5d51c5bab2b4ad7873aa68288bedec77eef80984406b9a77749" Jul 06 19:51:09.553867 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.553847 2566 scope.go:117] "RemoveContainer" containerID="5e51a134504a095a80da5435aaa4cb007359a76683089e1efeb49d4f64080c2b" Jul 06 19:51:09.562900 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.562878 2566 scope.go:117] "RemoveContainer" containerID="407169fb7e60c5d51c5bab2b4ad7873aa68288bedec77eef80984406b9a77749" Jul 06 19:51:09.563181 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:51:09.563159 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"407169fb7e60c5d51c5bab2b4ad7873aa68288bedec77eef80984406b9a77749\": container with ID starting with 407169fb7e60c5d51c5bab2b4ad7873aa68288bedec77eef80984406b9a77749 not found: ID does not exist" containerID="407169fb7e60c5d51c5bab2b4ad7873aa68288bedec77eef80984406b9a77749" Jul 06 19:51:09.563285 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.563187 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407169fb7e60c5d51c5bab2b4ad7873aa68288bedec77eef80984406b9a77749"} err="failed to get container status \"407169fb7e60c5d51c5bab2b4ad7873aa68288bedec77eef80984406b9a77749\": rpc error: code = NotFound desc = could not find container \"407169fb7e60c5d51c5bab2b4ad7873aa68288bedec77eef80984406b9a77749\": container with ID starting with 407169fb7e60c5d51c5bab2b4ad7873aa68288bedec77eef80984406b9a77749 not found: ID does not exist" Jul 06 19:51:09.563285 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.563203 2566 scope.go:117] "RemoveContainer" containerID="5e51a134504a095a80da5435aaa4cb007359a76683089e1efeb49d4f64080c2b" Jul 06 19:51:09.563464 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:51:09.563446 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e51a134504a095a80da5435aaa4cb007359a76683089e1efeb49d4f64080c2b\": container with ID starting with 5e51a134504a095a80da5435aaa4cb007359a76683089e1efeb49d4f64080c2b not found: ID does not exist" containerID="5e51a134504a095a80da5435aaa4cb007359a76683089e1efeb49d4f64080c2b" Jul 06 19:51:09.563506 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.563469 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e51a134504a095a80da5435aaa4cb007359a76683089e1efeb49d4f64080c2b"} err="failed to get container status \"5e51a134504a095a80da5435aaa4cb007359a76683089e1efeb49d4f64080c2b\": rpc error: code = NotFound desc = could not find container \"5e51a134504a095a80da5435aaa4cb007359a76683089e1efeb49d4f64080c2b\": container with ID starting with 5e51a134504a095a80da5435aaa4cb007359a76683089e1efeb49d4f64080c2b not found: ID does not exist" Jul 06 19:51:09.579921 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.579892 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm"] Jul 06 19:51:09.587452 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.587423 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f1ee26-openshift-default-c7ddf59cc-gk8qm"] Jul 06 19:51:09.764184 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:09.764148 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b25862e-3082-4019-abb3-724632fface5" path="/var/lib/kubelet/pods/7b25862e-3082-4019-abb3-724632fface5/volumes" Jul 06 19:51:10.198843 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:10.198806 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:10.349894 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:10.349860 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:10.448703 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:10.448616 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:10.545572 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:10.545525 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:10.878398 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:10.878290 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:10.975118 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:10.975084 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:14.721518 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:14.721481 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:15.204622 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.204591 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:15.482580 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.482496 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2"] Jul 06 19:51:15.483154 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.483132 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" containerName="istio-proxy" Jul 06 19:51:15.483202 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.483159 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" containerName="istio-proxy" Jul 06 19:51:15.483202 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.483173 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b25862e-3082-4019-abb3-724632fface5" containerName="istio-proxy" Jul 06 19:51:15.483202 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.483181 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b25862e-3082-4019-abb3-724632fface5" containerName="istio-proxy" Jul 06 19:51:15.483293 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.483202 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b25862e-3082-4019-abb3-724632fface5" containerName="istio-proxy" Jul 06 19:51:15.483293 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.483212 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b25862e-3082-4019-abb3-724632fface5" containerName="istio-proxy" Jul 06 19:51:15.483357 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.483338 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b25862e-3082-4019-abb3-724632fface5" containerName="istio-proxy" Jul 06 19:51:15.483390 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.483355 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" containerName="istio-proxy" Jul 06 19:51:15.483390 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.483365 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" containerName="istio-proxy" Jul 06 19:51:15.488459 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.488435 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.491214 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.491192 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d8b5cd-openshift-default-dockercfg-rdbqb\"" Jul 06 19:51:15.501599 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.501572 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2"] Jul 06 19:51:15.537140 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.537108 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:15.612370 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.612330 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-workload-socket\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.612600 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.612386 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-credential-socket\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.612600 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.612414 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-workload-certs\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.612600 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.612443 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-data\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.612600 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.612506 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c8bb05-3876-4906-831c-592145d9c7f3-istiod-ca-cert\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.612600 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.612575 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-podinfo\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.612909 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.612668 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-envoy\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.612909 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.612764 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-token\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.612909 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.612800 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n84zk\" (UniqueName: \"kubernetes.io/projected/a5c8bb05-3876-4906-831c-592145d9c7f3-kube-api-access-n84zk\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.713695 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.713660 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-workload-certs\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.713695 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.713700 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-data\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.713957 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.713753 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c8bb05-3876-4906-831c-592145d9c7f3-istiod-ca-cert\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.713957 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.713778 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-podinfo\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.713957 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.713812 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-envoy\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.713957 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.713847 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-token\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.713957 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.713877 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n84zk\" (UniqueName: \"kubernetes.io/projected/a5c8bb05-3876-4906-831c-592145d9c7f3-kube-api-access-n84zk\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.713957 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.713957 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-workload-socket\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.714274 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.713991 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-credential-socket\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.714274 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.714147 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-workload-certs\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.714274 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.714165 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-data\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.714436 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.714380 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-credential-socket\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.714494 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.714466 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-workload-socket\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.714931 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.714911 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c8bb05-3876-4906-831c-592145d9c7f3-istiod-ca-cert\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.716383 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.716361 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-envoy\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.716564 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.716547 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-podinfo\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.728646 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.728604 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n84zk\" (UniqueName: \"kubernetes.io/projected/a5c8bb05-3876-4906-831c-592145d9c7f3-kube-api-access-n84zk\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.729118 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.728768 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-token\") pod \"e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:15.801495 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:15.801404 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:16.004055 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:16.004031 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2"] Jul 06 19:51:16.006361 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:51:16.006331 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c8bb05_3876_4906_831c_592145d9c7f3.slice/crio-414813a8c91ae02c4606e20dadd6bc9d550f89d6c23bf2e7f8d9f61cade4ebcf WatchSource:0}: Error finding container 414813a8c91ae02c4606e20dadd6bc9d550f89d6c23bf2e7f8d9f61cade4ebcf: Status 404 returned error can't find the container with id 414813a8c91ae02c4606e20dadd6bc9d550f89d6c23bf2e7f8d9f61cade4ebcf Jul 06 19:51:16.008548 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:16.008514 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:51:16.008655 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:16.008583 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:51:16.008655 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:16.008615 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:51:16.194934 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:16.194895 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:16.582447 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:16.582410 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" event={"ID":"a5c8bb05-3876-4906-831c-592145d9c7f3","Type":"ContainerStarted","Data":"d4d921e03b7c466ec4096f689f6f435a9944ba2ba8c387bddec72da29c871105"} Jul 06 19:51:16.582635 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:16.582466 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" event={"ID":"a5c8bb05-3876-4906-831c-592145d9c7f3","Type":"ContainerStarted","Data":"414813a8c91ae02c4606e20dadd6bc9d550f89d6c23bf2e7f8d9f61cade4ebcf"} Jul 06 19:51:16.614944 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:16.614894 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" podStartSLOduration=1.614877619 podStartE2EDuration="1.614877619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:16.611474966 +0000 UTC m=+2373.507364497" watchObservedRunningTime="2026-07-06 19:51:16.614877619 +0000 UTC m=+2373.510767152" Jul 06 19:51:16.802507 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:16.802459 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:17.807040 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:17.807011 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:18.593965 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:18.593935 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:18.594957 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:18.594935 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:18.750343 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:18.750313 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:18.857244 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:18.857156 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:18.954939 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:18.954913 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:19.114637 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:19.114551 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:19.618861 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:19.618827 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:19.740964 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:19.740937 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:23.211594 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:23.211559 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:26.120683 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:26.120648 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:30.681774 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:30.681704 2566 generic.go:356] "Generic (PLEG): container finished" podID="a5c8bb05-3876-4906-831c-592145d9c7f3" containerID="d4d921e03b7c466ec4096f689f6f435a9944ba2ba8c387bddec72da29c871105" exitCode=0 Jul 06 19:51:30.682195 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:30.681775 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" event={"ID":"a5c8bb05-3876-4906-831c-592145d9c7f3","Type":"ContainerDied","Data":"d4d921e03b7c466ec4096f689f6f435a9944ba2ba8c387bddec72da29c871105"} Jul 06 19:51:30.682240 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:30.682213 2566 scope.go:117] "RemoveContainer" containerID="d4d921e03b7c466ec4096f689f6f435a9944ba2ba8c387bddec72da29c871105" Jul 06 19:51:30.683432 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:30.683399 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:51:30.683511 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:30.683491 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:51:30.683554 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:30.683536 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:51:31.527111 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:31.527080 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:31.689033 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:31.688995 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" event={"ID":"a5c8bb05-3876-4906-831c-592145d9c7f3","Type":"ContainerStarted","Data":"0f78da584b8a642b69e3108e531a372b5f5fdb643745a1f891983dc992d142c0"} Jul 06 19:51:31.802547 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:31.802458 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:31.803451 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:31.803431 2566 patch_prober.go:28] interesting pod/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 06 19:51:31.803563 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:31.803470 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" podUID="a5c8bb05-3876-4906-831c-592145d9c7f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 06 19:51:31.923320 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:31.923287 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:32.019551 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:32.019518 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:32.121282 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:32.121201 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:32.294505 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:32.294470 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:32.801880 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:32.801842 2566 patch_prober.go:28] interesting pod/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 06 19:51:32.802256 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:32.801895 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" podUID="a5c8bb05-3876-4906-831c-592145d9c7f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 06 19:51:33.593857 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:33.593818 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:33.802037 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:33.802001 2566 patch_prober.go:28] interesting pod/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 06 19:51:33.802446 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:33.802054 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" podUID="a5c8bb05-3876-4906-831c-592145d9c7f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 06 19:51:34.802864 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:34.802824 2566 patch_prober.go:28] interesting pod/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 06 19:51:34.803252 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:34.802890 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" podUID="a5c8bb05-3876-4906-831c-592145d9c7f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 06 19:51:35.884544 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:35.884511 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:35.886245 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:35.886213 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:38.363595 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:38.363563 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:38.502873 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:38.502838 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:38.751539 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:38.751507 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:38.959516 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:38.959484 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:39.179617 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:39.179578 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:39.595435 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:39.595341 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:39.995290 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:39.995255 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:40.227289 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:40.227248 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:40.546350 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:40.546322 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:43.802451 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:43.802421 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:51:43.830754 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:43.830701 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:51:45.489199 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:45.489163 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:45.588848 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:45.588811 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:45.764611 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:45.764527 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2"] Jul 06 19:51:45.764813 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:45.764787 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" podUID="a5c8bb05-3876-4906-831c-592145d9c7f3" containerName="istio-proxy" containerID="cri-o://0f78da584b8a642b69e3108e531a372b5f5fdb643745a1f891983dc992d142c0" gracePeriod=30 Jul 06 19:51:45.799180 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:45.799135 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:46.029245 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:46.029157 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:46.393069 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:46.392979 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:46.573958 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:46.573914 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:46.918302 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:46.918272 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:47.206117 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:47.206031 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:49.504107 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:49.504063 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:49.658630 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:49.658603 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:49.771543 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:49.771457 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:49.883179 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:49.883145 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:50.452735 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:50.452691 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:50.552503 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:50.552466 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:50.887135 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:50.887096 2566 patch_prober.go:28] interesting pod/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 06 19:51:50.887258 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:50.887179 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" podUID="a5c8bb05-3876-4906-831c-592145d9c7f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 06 19:51:51.030851 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.030781 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:51.146285 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.146257 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c8bb05-3876-4906-831c-592145d9c7f3-istiod-ca-cert\") pod \"a5c8bb05-3876-4906-831c-592145d9c7f3\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " Jul 06 19:51:51.146285 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.146290 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-workload-socket\") pod \"a5c8bb05-3876-4906-831c-592145d9c7f3\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " Jul 06 19:51:51.146548 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.146306 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-envoy\") pod \"a5c8bb05-3876-4906-831c-592145d9c7f3\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " Jul 06 19:51:51.146548 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.146321 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-token\") pod \"a5c8bb05-3876-4906-831c-592145d9c7f3\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " Jul 06 19:51:51.146548 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.146350 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-credential-socket\") pod \"a5c8bb05-3876-4906-831c-592145d9c7f3\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " Jul 06 19:51:51.146548 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.146366 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n84zk\" (UniqueName: \"kubernetes.io/projected/a5c8bb05-3876-4906-831c-592145d9c7f3-kube-api-access-n84zk\") pod \"a5c8bb05-3876-4906-831c-592145d9c7f3\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " Jul 06 19:51:51.146548 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.146404 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-data\") pod \"a5c8bb05-3876-4906-831c-592145d9c7f3\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " Jul 06 19:51:51.146548 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.146429 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-podinfo\") pod \"a5c8bb05-3876-4906-831c-592145d9c7f3\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " Jul 06 19:51:51.146548 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.146550 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-workload-certs\") pod \"a5c8bb05-3876-4906-831c-592145d9c7f3\" (UID: \"a5c8bb05-3876-4906-831c-592145d9c7f3\") " Jul 06 19:51:51.146962 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.146696 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a5c8bb05-3876-4906-831c-592145d9c7f3" (UID: "a5c8bb05-3876-4906-831c-592145d9c7f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:51.146962 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.146708 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a5c8bb05-3876-4906-831c-592145d9c7f3" (UID: "a5c8bb05-3876-4906-831c-592145d9c7f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:51.146962 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.146835 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c8bb05-3876-4906-831c-592145d9c7f3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a5c8bb05-3876-4906-831c-592145d9c7f3" (UID: "a5c8bb05-3876-4906-831c-592145d9c7f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:51:51.147133 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.146964 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a5c8bb05-3876-4906-831c-592145d9c7f3" (UID: "a5c8bb05-3876-4906-831c-592145d9c7f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:51.147133 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.147084 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c8bb05-3876-4906-831c-592145d9c7f3-istiod-ca-cert\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:51.147133 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.147108 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-workload-socket\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:51.147133 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.147122 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-credential-socket\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:51.147353 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.147134 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-workload-certs\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:51.147579 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.147551 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-data" (OuterVolumeSpecName: "istio-data") pod "a5c8bb05-3876-4906-831c-592145d9c7f3" (UID: "a5c8bb05-3876-4906-831c-592145d9c7f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:51.148771 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.148713 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a5c8bb05-3876-4906-831c-592145d9c7f3" (UID: "a5c8bb05-3876-4906-831c-592145d9c7f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:51.148870 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.148767 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c8bb05-3876-4906-831c-592145d9c7f3-kube-api-access-n84zk" (OuterVolumeSpecName: "kube-api-access-n84zk") pod "a5c8bb05-3876-4906-831c-592145d9c7f3" (UID: "a5c8bb05-3876-4906-831c-592145d9c7f3"). InnerVolumeSpecName "kube-api-access-n84zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:51:51.148913 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.148868 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-token" (OuterVolumeSpecName: "istio-token") pod "a5c8bb05-3876-4906-831c-592145d9c7f3" (UID: "a5c8bb05-3876-4906-831c-592145d9c7f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:51:51.148983 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.148967 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a5c8bb05-3876-4906-831c-592145d9c7f3" (UID: "a5c8bb05-3876-4906-831c-592145d9c7f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:51:51.247909 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.247871 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-envoy\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:51.247909 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.247902 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-token\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:51.247909 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.247912 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n84zk\" (UniqueName: \"kubernetes.io/projected/a5c8bb05-3876-4906-831c-592145d9c7f3-kube-api-access-n84zk\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:51.247909 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.247921 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-data\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:51.248181 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.247930 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c8bb05-3876-4906-831c-592145d9c7f3-istio-podinfo\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:51:51.804875 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.804841 2566 generic.go:356] "Generic (PLEG): container finished" podID="a5c8bb05-3876-4906-831c-592145d9c7f3" containerID="0f78da584b8a642b69e3108e531a372b5f5fdb643745a1f891983dc992d142c0" exitCode=0 Jul 06 19:51:51.805341 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.804911 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" Jul 06 19:51:51.805341 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.804914 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" event={"ID":"a5c8bb05-3876-4906-831c-592145d9c7f3","Type":"ContainerDied","Data":"0f78da584b8a642b69e3108e531a372b5f5fdb643745a1f891983dc992d142c0"} Jul 06 19:51:51.805341 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.805014 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2" event={"ID":"a5c8bb05-3876-4906-831c-592145d9c7f3","Type":"ContainerDied","Data":"414813a8c91ae02c4606e20dadd6bc9d550f89d6c23bf2e7f8d9f61cade4ebcf"} Jul 06 19:51:51.805341 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.805046 2566 scope.go:117] "RemoveContainer" containerID="0f78da584b8a642b69e3108e531a372b5f5fdb643745a1f891983dc992d142c0" Jul 06 19:51:51.814339 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.814320 2566 scope.go:117] "RemoveContainer" containerID="d4d921e03b7c466ec4096f689f6f435a9944ba2ba8c387bddec72da29c871105" Jul 06 19:51:51.822645 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.822623 2566 scope.go:117] "RemoveContainer" containerID="0f78da584b8a642b69e3108e531a372b5f5fdb643745a1f891983dc992d142c0" Jul 06 19:51:51.823043 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:51:51.823017 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f78da584b8a642b69e3108e531a372b5f5fdb643745a1f891983dc992d142c0\": container with ID starting with 0f78da584b8a642b69e3108e531a372b5f5fdb643745a1f891983dc992d142c0 not found: ID does not exist" containerID="0f78da584b8a642b69e3108e531a372b5f5fdb643745a1f891983dc992d142c0" Jul 06 19:51:51.823218 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.823049 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f78da584b8a642b69e3108e531a372b5f5fdb643745a1f891983dc992d142c0"} err="failed to get container status \"0f78da584b8a642b69e3108e531a372b5f5fdb643745a1f891983dc992d142c0\": rpc error: code = NotFound desc = could not find container \"0f78da584b8a642b69e3108e531a372b5f5fdb643745a1f891983dc992d142c0\": container with ID starting with 0f78da584b8a642b69e3108e531a372b5f5fdb643745a1f891983dc992d142c0 not found: ID does not exist" Jul 06 19:51:51.823218 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.823068 2566 scope.go:117] "RemoveContainer" containerID="d4d921e03b7c466ec4096f689f6f435a9944ba2ba8c387bddec72da29c871105" Jul 06 19:51:51.823390 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:51:51.823321 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d921e03b7c466ec4096f689f6f435a9944ba2ba8c387bddec72da29c871105\": container with ID starting with d4d921e03b7c466ec4096f689f6f435a9944ba2ba8c387bddec72da29c871105 not found: ID does not exist" containerID="d4d921e03b7c466ec4096f689f6f435a9944ba2ba8c387bddec72da29c871105" Jul 06 19:51:51.823390 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.823348 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d921e03b7c466ec4096f689f6f435a9944ba2ba8c387bddec72da29c871105"} err="failed to get container status \"d4d921e03b7c466ec4096f689f6f435a9944ba2ba8c387bddec72da29c871105\": rpc error: code = NotFound desc = could not find container \"d4d921e03b7c466ec4096f689f6f435a9944ba2ba8c387bddec72da29c871105\": container with ID starting with d4d921e03b7c466ec4096f689f6f435a9944ba2ba8c387bddec72da29c871105 not found: ID does not exist" Jul 06 19:51:51.826452 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.826430 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2"] Jul 06 19:51:51.834053 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:51.834032 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d8b5cd-openshift-default-8678cbbcc7-n2lw2"] Jul 06 19:51:53.764462 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:53.764431 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5c8bb05-3876-4906-831c-592145d9c7f3" path="/var/lib/kubelet/pods/a5c8bb05-3876-4906-831c-592145d9c7f3/volumes" Jul 06 19:51:54.005764 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.005705 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:54.365784 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.365749 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:54.654874 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.654790 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj"] Jul 06 19:51:54.655178 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.655166 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" containerName="istio-proxy" Jul 06 19:51:54.655178 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.655179 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf2a33e7-f42f-4a34-8a59-64317fcc4f5b" containerName="istio-proxy" Jul 06 19:51:54.655269 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.655192 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5c8bb05-3876-4906-831c-592145d9c7f3" containerName="istio-proxy" Jul 06 19:51:54.655269 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.655227 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5c8bb05-3876-4906-831c-592145d9c7f3" containerName="istio-proxy" Jul 06 19:51:54.655269 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.655248 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5c8bb05-3876-4906-831c-592145d9c7f3" containerName="istio-proxy" Jul 06 19:51:54.655269 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.655256 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5c8bb05-3876-4906-831c-592145d9c7f3" containerName="istio-proxy" Jul 06 19:51:54.655453 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.655352 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b25862e-3082-4019-abb3-724632fface5" containerName="istio-proxy" Jul 06 19:51:54.655453 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.655364 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5c8bb05-3876-4906-831c-592145d9c7f3" containerName="istio-proxy" Jul 06 19:51:54.655453 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.655375 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5c8bb05-3876-4906-831c-592145d9c7f3" containerName="istio-proxy" Jul 06 19:51:54.659935 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.659916 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.663882 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.663864 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-c6207b-openshift-default-dockercfg-6ggln\"" Jul 06 19:51:54.680288 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.680259 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj"] Jul 06 19:51:54.705407 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.705377 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:54.781558 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.781510 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-credential-socket\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.782332 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.781566 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-envoy\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.782332 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.781597 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6plz4\" (UniqueName: \"kubernetes.io/projected/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-kube-api-access-6plz4\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.782332 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.781654 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-data\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.782332 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.781799 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-workload-socket\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.782332 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.781833 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istiod-ca-cert\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.782332 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.781906 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-workload-certs\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.782332 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.781932 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-token\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.782332 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.781970 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-podinfo\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.882535 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.882498 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-data\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.882763 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.882562 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-workload-socket\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.882763 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.882579 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istiod-ca-cert\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.882763 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.882746 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-workload-certs\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.882960 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.882815 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-token\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.882960 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.882864 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-podinfo\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.883064 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.883000 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-credential-socket\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.883064 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.883016 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-data\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.883064 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.883047 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-envoy\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.883220 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.883077 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6plz4\" (UniqueName: \"kubernetes.io/projected/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-kube-api-access-6plz4\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.883292 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.883276 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-credential-socket\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.883360 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.883338 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istiod-ca-cert\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.883416 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.883400 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-workload-certs\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.883416 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.883399 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-workload-socket\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.885242 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.885218 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-envoy\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.885571 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.885552 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-podinfo\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.892022 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.891990 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-token\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.892150 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.892133 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6plz4\" (UniqueName: \"kubernetes.io/projected/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-kube-api-access-6plz4\") pod \"e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:54.970868 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:54.970825 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:55.129369 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:55.129230 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj"] Jul 06 19:51:55.130164 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:51:55.130141 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c41ef09_8de4_478c_bf7f_af9a1292bb2d.slice/crio-8812f7894a788412a0b430302f6653c3258da59b708dcc01ad4ca764ccee6a67 WatchSource:0}: Error finding container 8812f7894a788412a0b430302f6653c3258da59b708dcc01ad4ca764ccee6a67: Status 404 returned error can't find the container with id 8812f7894a788412a0b430302f6653c3258da59b708dcc01ad4ca764ccee6a67 Jul 06 19:51:55.132891 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:55.132847 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:51:55.133014 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:55.132938 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:51:55.133014 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:55.132980 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:51:55.434315 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:55.434278 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:55.829866 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:55.829785 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" event={"ID":"1c41ef09-8de4-478c-bf7f-af9a1292bb2d","Type":"ContainerStarted","Data":"f687a08847b295cce0fdf8758772692735523fa0c0c98d0db37636a1e24c835c"} Jul 06 19:51:55.829866 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:55.829838 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" event={"ID":"1c41ef09-8de4-478c-bf7f-af9a1292bb2d","Type":"ContainerStarted","Data":"8812f7894a788412a0b430302f6653c3258da59b708dcc01ad4ca764ccee6a67"} Jul 06 19:51:55.862191 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:55.862139 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" podStartSLOduration=1.8621231360000001 podStartE2EDuration="1.862123136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:55.860827224 +0000 UTC m=+2412.756716757" watchObservedRunningTime="2026-07-06 19:51:55.862123136 +0000 UTC m=+2412.758012667" Jul 06 19:51:55.971889 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:55.971850 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:56.976870 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:56.976842 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:57.841334 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:57.841298 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:57.842347 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:57.842325 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:51:58.068295 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:58.068267 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:58.417230 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:58.417198 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:58.755261 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:58.755169 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:59.117842 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:59.117757 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:59.636692 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:59.636656 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:59.735278 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:51:59.735243 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:02.432329 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:02.432299 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:11.362440 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:11.362400 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:11.462039 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:11.462005 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:12.710392 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:12.710311 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:13.882910 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:13.882861 2566 patch_prober.go:28] interesting pod/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 19:52:13.883460 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:13.882930 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" podUID="1c41ef09-8de4-478c-bf7f-af9a1292bb2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 19:52:15.940666 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:15.940628 2566 generic.go:356] "Generic (PLEG): container finished" podID="1c41ef09-8de4-478c-bf7f-af9a1292bb2d" containerID="f687a08847b295cce0fdf8758772692735523fa0c0c98d0db37636a1e24c835c" exitCode=0 Jul 06 19:52:15.941176 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:15.940695 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" event={"ID":"1c41ef09-8de4-478c-bf7f-af9a1292bb2d","Type":"ContainerDied","Data":"f687a08847b295cce0fdf8758772692735523fa0c0c98d0db37636a1e24c835c"} Jul 06 19:52:15.941176 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:15.941126 2566 scope.go:117] "RemoveContainer" containerID="f687a08847b295cce0fdf8758772692735523fa0c0c98d0db37636a1e24c835c" Jul 06 19:52:15.942163 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:15.942126 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:52:15.942262 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:15.942204 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:52:15.942262 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:15.942238 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:52:16.955116 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:16.955077 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" event={"ID":"1c41ef09-8de4-478c-bf7f-af9a1292bb2d","Type":"ContainerStarted","Data":"251696666b3fdbc45f6276b374a69f4d1e009cdaf3b9181ae15eb737f0bae9e0"} Jul 06 19:52:16.971538 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:16.971504 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:52:17.971376 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:17.971342 2566 patch_prober.go:28] interesting pod/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 06 19:52:17.971789 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:17.971393 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" podUID="1c41ef09-8de4-478c-bf7f-af9a1292bb2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 06 19:52:18.972135 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:18.972101 2566 patch_prober.go:28] interesting pod/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 06 19:52:18.972574 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:18.972160 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" podUID="1c41ef09-8de4-478c-bf7f-af9a1292bb2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 06 19:52:19.971770 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:19.971704 2566 patch_prober.go:28] interesting pod/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 06 19:52:19.971982 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:19.971793 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" podUID="1c41ef09-8de4-478c-bf7f-af9a1292bb2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 06 19:52:20.551608 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:20.551544 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:20.645907 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:20.645866 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:20.975993 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:20.975960 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:52:20.976410 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:20.976391 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:52:20.977128 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:20.977104 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:52:28.715554 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.715510 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc"] Jul 06 19:52:28.719088 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.719064 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:28.721913 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.721888 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6207b\"/\"kube-root-ca.crt\"" Jul 06 19:52:28.722581 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.722564 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6207b\"/\"default-dockercfg-jzxfb\"" Jul 06 19:52:28.722648 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.722586 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6207b\"/\"auth-test-model-c6207b-kserve-self-signed-certs\"" Jul 06 19:52:28.722697 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.722685 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6207b\"/\"openshift-service-ca.crt\"" Jul 06 19:52:28.731174 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.731143 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc"] Jul 06 19:52:28.803913 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.803882 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:28.806100 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.806079 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-dshm\") pod \"auth-test-model-c6207b-kserve-7b49b99d59-kn6cc\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:28.806194 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.806130 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td959\" (UniqueName: \"kubernetes.io/projected/510bcd11-80ef-4a06-ac59-be35cee3088a-kube-api-access-td959\") pod \"auth-test-model-c6207b-kserve-7b49b99d59-kn6cc\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:28.806241 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.806200 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510bcd11-80ef-4a06-ac59-be35cee3088a-tls-certs\") pod \"auth-test-model-c6207b-kserve-7b49b99d59-kn6cc\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:28.806276 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.806259 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-home\") pod \"auth-test-model-c6207b-kserve-7b49b99d59-kn6cc\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:28.806311 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.806293 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-model-cache\") pod \"auth-test-model-c6207b-kserve-7b49b99d59-kn6cc\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:28.806345 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.806316 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-tmp-dir\") pod \"auth-test-model-c6207b-kserve-7b49b99d59-kn6cc\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:28.906956 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.906921 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-dshm\") pod \"auth-test-model-c6207b-kserve-7b49b99d59-kn6cc\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:28.907163 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.906981 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td959\" (UniqueName: \"kubernetes.io/projected/510bcd11-80ef-4a06-ac59-be35cee3088a-kube-api-access-td959\") pod \"auth-test-model-c6207b-kserve-7b49b99d59-kn6cc\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:28.907163 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.907025 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510bcd11-80ef-4a06-ac59-be35cee3088a-tls-certs\") pod \"auth-test-model-c6207b-kserve-7b49b99d59-kn6cc\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:28.907163 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.907071 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-home\") pod \"auth-test-model-c6207b-kserve-7b49b99d59-kn6cc\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:28.907163 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.907108 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-model-cache\") pod \"auth-test-model-c6207b-kserve-7b49b99d59-kn6cc\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:28.907163 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.907129 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-tmp-dir\") pod \"auth-test-model-c6207b-kserve-7b49b99d59-kn6cc\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:28.907505 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.907474 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-home\") pod \"auth-test-model-c6207b-kserve-7b49b99d59-kn6cc\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:28.907505 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.907484 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-model-cache\") pod \"auth-test-model-c6207b-kserve-7b49b99d59-kn6cc\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:28.907698 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.907584 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-tmp-dir\") pod \"auth-test-model-c6207b-kserve-7b49b99d59-kn6cc\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:28.909283 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.909265 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-dshm\") pod \"auth-test-model-c6207b-kserve-7b49b99d59-kn6cc\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:28.909585 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.909569 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510bcd11-80ef-4a06-ac59-be35cee3088a-tls-certs\") pod \"auth-test-model-c6207b-kserve-7b49b99d59-kn6cc\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:28.917003 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:28.916978 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td959\" (UniqueName: \"kubernetes.io/projected/510bcd11-80ef-4a06-ac59-be35cee3088a-kube-api-access-td959\") pod \"auth-test-model-c6207b-kserve-7b49b99d59-kn6cc\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:29.029342 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:29.029239 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:29.176121 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:29.176094 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc"] Jul 06 19:52:29.177756 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:52:29.177691 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod510bcd11_80ef_4a06_ac59_be35cee3088a.slice/crio-0a473cfad5c82ec3e63540ef84c08d70f34f665d26dc2c192bf9bc54afdf8d59 WatchSource:0}: Error finding container 0a473cfad5c82ec3e63540ef84c08d70f34f665d26dc2c192bf9bc54afdf8d59: Status 404 returned error can't find the container with id 0a473cfad5c82ec3e63540ef84c08d70f34f665d26dc2c192bf9bc54afdf8d59 Jul 06 19:52:30.031117 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:30.031081 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" event={"ID":"510bcd11-80ef-4a06-ac59-be35cee3088a","Type":"ContainerStarted","Data":"2febf7473dceafd0599fd1a41758d132c657a64692a373cbf4e97b4c75161716"} Jul 06 19:52:30.031117 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:30.031124 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" event={"ID":"510bcd11-80ef-4a06-ac59-be35cee3088a","Type":"ContainerStarted","Data":"0a473cfad5c82ec3e63540ef84c08d70f34f665d26dc2c192bf9bc54afdf8d59"} Jul 06 19:52:30.050317 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:30.050267 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" podStartSLOduration=2.05025093 podStartE2EDuration="2.05025093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:30.048684262 +0000 UTC m=+2446.944573818" watchObservedRunningTime="2026-07-06 19:52:30.05025093 +0000 UTC m=+2446.946140461" Jul 06 19:52:39.030051 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:39.030013 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:39.030488 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:39.030277 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:39.042861 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:39.042831 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:52:39.095055 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:52:39.095017 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:53:03.516207 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:53:03.516176 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:18.079198 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:53:18.079161 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:27.704899 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:27.704867 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:28.517379 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:28.517344 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc"] Jul 06 19:54:28.548202 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:28.548166 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:35.304064 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:35.304022 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:43.539132 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.539093 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" podUID="510bcd11-80ef-4a06-ac59-be35cee3088a" containerName="main" containerID="cri-o://2febf7473dceafd0599fd1a41758d132c657a64692a373cbf4e97b4c75161716" gracePeriod=45 Jul 06 19:54:43.772849 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.772810 2566 generic.go:356] "Generic (PLEG): container finished" podID="510bcd11-80ef-4a06-ac59-be35cee3088a" containerID="2febf7473dceafd0599fd1a41758d132c657a64692a373cbf4e97b4c75161716" exitCode=0 Jul 06 19:54:43.773021 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.772896 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" event={"ID":"510bcd11-80ef-4a06-ac59-be35cee3088a","Type":"ContainerDied","Data":"2febf7473dceafd0599fd1a41758d132c657a64692a373cbf4e97b4c75161716"} Jul 06 19:54:43.806317 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.806289 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:54:43.851015 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.850985 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-dshm\") pod \"510bcd11-80ef-4a06-ac59-be35cee3088a\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " Jul 06 19:54:43.851197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.851044 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td959\" (UniqueName: \"kubernetes.io/projected/510bcd11-80ef-4a06-ac59-be35cee3088a-kube-api-access-td959\") pod \"510bcd11-80ef-4a06-ac59-be35cee3088a\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " Jul 06 19:54:43.851197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.851103 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-home\") pod \"510bcd11-80ef-4a06-ac59-be35cee3088a\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " Jul 06 19:54:43.851197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.851124 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510bcd11-80ef-4a06-ac59-be35cee3088a-tls-certs\") pod \"510bcd11-80ef-4a06-ac59-be35cee3088a\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " Jul 06 19:54:43.851197 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.851166 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-tmp-dir\") pod \"510bcd11-80ef-4a06-ac59-be35cee3088a\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " Jul 06 19:54:43.851406 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.851215 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-model-cache\") pod \"510bcd11-80ef-4a06-ac59-be35cee3088a\" (UID: \"510bcd11-80ef-4a06-ac59-be35cee3088a\") " Jul 06 19:54:43.851467 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.851412 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-home" (OuterVolumeSpecName: "home") pod "510bcd11-80ef-4a06-ac59-be35cee3088a" (UID: "510bcd11-80ef-4a06-ac59-be35cee3088a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:43.851575 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.851539 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "510bcd11-80ef-4a06-ac59-be35cee3088a" (UID: "510bcd11-80ef-4a06-ac59-be35cee3088a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:43.851868 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.851662 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-model-cache" (OuterVolumeSpecName: "model-cache") pod "510bcd11-80ef-4a06-ac59-be35cee3088a" (UID: "510bcd11-80ef-4a06-ac59-be35cee3088a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:43.853451 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.853425 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/510bcd11-80ef-4a06-ac59-be35cee3088a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "510bcd11-80ef-4a06-ac59-be35cee3088a" (UID: "510bcd11-80ef-4a06-ac59-be35cee3088a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:54:43.853451 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.853438 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510bcd11-80ef-4a06-ac59-be35cee3088a-kube-api-access-td959" (OuterVolumeSpecName: "kube-api-access-td959") pod "510bcd11-80ef-4a06-ac59-be35cee3088a" (UID: "510bcd11-80ef-4a06-ac59-be35cee3088a"). InnerVolumeSpecName "kube-api-access-td959". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:54:43.853599 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.853579 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-dshm" (OuterVolumeSpecName: "dshm") pod "510bcd11-80ef-4a06-ac59-be35cee3088a" (UID: "510bcd11-80ef-4a06-ac59-be35cee3088a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:43.952746 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.952686 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-dshm\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:54:43.952746 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.952714 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td959\" (UniqueName: \"kubernetes.io/projected/510bcd11-80ef-4a06-ac59-be35cee3088a-kube-api-access-td959\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:54:43.952969 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.952763 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-home\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:54:43.952969 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.952779 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510bcd11-80ef-4a06-ac59-be35cee3088a-tls-certs\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:54:43.952969 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.952790 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-tmp-dir\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:54:43.952969 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:43.952801 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/510bcd11-80ef-4a06-ac59-be35cee3088a-model-cache\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:54:44.780310 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:44.780277 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" Jul 06 19:54:44.780805 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:44.780305 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc" event={"ID":"510bcd11-80ef-4a06-ac59-be35cee3088a","Type":"ContainerDied","Data":"0a473cfad5c82ec3e63540ef84c08d70f34f665d26dc2c192bf9bc54afdf8d59"} Jul 06 19:54:44.780805 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:44.780360 2566 scope.go:117] "RemoveContainer" containerID="2febf7473dceafd0599fd1a41758d132c657a64692a373cbf4e97b4c75161716" Jul 06 19:54:44.802061 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:44.802032 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc"] Jul 06 19:54:44.806211 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:44.806188 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c6207b/auth-test-model-c6207b-kserve-7b49b99d59-kn6cc"] Jul 06 19:54:45.763786 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:54:45.763751 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="510bcd11-80ef-4a06-ac59-be35cee3088a" path="/var/lib/kubelet/pods/510bcd11-80ef-4a06-ac59-be35cee3088a/volumes" Jul 06 19:55:08.921769 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:08.921658 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r"] Jul 06 19:55:08.922237 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:08.922122 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="510bcd11-80ef-4a06-ac59-be35cee3088a" containerName="main" Jul 06 19:55:08.922237 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:08.922137 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="510bcd11-80ef-4a06-ac59-be35cee3088a" containerName="main" Jul 06 19:55:08.922237 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:08.922210 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="510bcd11-80ef-4a06-ac59-be35cee3088a" containerName="main" Jul 06 19:55:08.925320 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:08.925301 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:08.930869 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:08.930844 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6207b\"/\"kube-root-ca.crt\"" Jul 06 19:55:08.931949 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:08.931927 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6207b\"/\"sub-test-model-c6207b-kserve-self-signed-certs\"" Jul 06 19:55:08.932099 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:08.932014 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6207b\"/\"default-dockercfg-jzxfb\"" Jul 06 19:55:08.932099 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:08.932042 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6207b\"/\"openshift-service-ca.crt\"" Jul 06 19:55:08.958868 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:08.958837 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r"] Jul 06 19:55:08.966063 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:08.966026 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:09.082789 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.082714 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-model-cache\") pod \"sub-test-model-c6207b-kserve-766f8fcc9-87z6r\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:09.082992 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.082804 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t45ph\" (UniqueName: \"kubernetes.io/projected/6dab534c-58d8-4a41-82f3-605234244dd3-kube-api-access-t45ph\") pod \"sub-test-model-c6207b-kserve-766f8fcc9-87z6r\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:09.082992 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.082921 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-tmp-dir\") pod \"sub-test-model-c6207b-kserve-766f8fcc9-87z6r\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:09.082992 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.082982 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-home\") pod \"sub-test-model-c6207b-kserve-766f8fcc9-87z6r\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:09.083126 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.083020 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dab534c-58d8-4a41-82f3-605234244dd3-tls-certs\") pod \"sub-test-model-c6207b-kserve-766f8fcc9-87z6r\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:09.083126 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.083050 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-dshm\") pod \"sub-test-model-c6207b-kserve-766f8fcc9-87z6r\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:09.184258 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.184163 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-tmp-dir\") pod \"sub-test-model-c6207b-kserve-766f8fcc9-87z6r\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:09.184258 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.184204 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-home\") pod \"sub-test-model-c6207b-kserve-766f8fcc9-87z6r\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:09.184258 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.184230 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dab534c-58d8-4a41-82f3-605234244dd3-tls-certs\") pod \"sub-test-model-c6207b-kserve-766f8fcc9-87z6r\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:09.184258 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.184248 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-dshm\") pod \"sub-test-model-c6207b-kserve-766f8fcc9-87z6r\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:09.184559 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.184496 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-model-cache\") pod \"sub-test-model-c6207b-kserve-766f8fcc9-87z6r\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:09.184600 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.184556 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t45ph\" (UniqueName: \"kubernetes.io/projected/6dab534c-58d8-4a41-82f3-605234244dd3-kube-api-access-t45ph\") pod \"sub-test-model-c6207b-kserve-766f8fcc9-87z6r\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:09.184709 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.184686 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-home\") pod \"sub-test-model-c6207b-kserve-766f8fcc9-87z6r\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:09.184805 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.184715 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-tmp-dir\") pod \"sub-test-model-c6207b-kserve-766f8fcc9-87z6r\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:09.184866 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.184826 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-model-cache\") pod \"sub-test-model-c6207b-kserve-766f8fcc9-87z6r\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:09.186481 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.186461 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-dshm\") pod \"sub-test-model-c6207b-kserve-766f8fcc9-87z6r\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:09.186795 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.186777 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dab534c-58d8-4a41-82f3-605234244dd3-tls-certs\") pod \"sub-test-model-c6207b-kserve-766f8fcc9-87z6r\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:09.193292 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.193272 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t45ph\" (UniqueName: \"kubernetes.io/projected/6dab534c-58d8-4a41-82f3-605234244dd3-kube-api-access-t45ph\") pod \"sub-test-model-c6207b-kserve-766f8fcc9-87z6r\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:09.235672 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.235636 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:09.380732 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.380695 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r"] Jul 06 19:55:09.383254 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:55:09.383229 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6dab534c_58d8_4a41_82f3_605234244dd3.slice/crio-ad9d3aafabbe4aa1dc486a5a428699eb337b5538745fc024cf99692e6949c7e3 WatchSource:0}: Error finding container ad9d3aafabbe4aa1dc486a5a428699eb337b5538745fc024cf99692e6949c7e3: Status 404 returned error can't find the container with id ad9d3aafabbe4aa1dc486a5a428699eb337b5538745fc024cf99692e6949c7e3 Jul 06 19:55:09.923919 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.923879 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" event={"ID":"6dab534c-58d8-4a41-82f3-605234244dd3","Type":"ContainerStarted","Data":"a9a22943214aa528cca7c399da01db0cb393a556178837f634849214cbe7337b"} Jul 06 19:55:09.923919 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.923923 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" event={"ID":"6dab534c-58d8-4a41-82f3-605234244dd3","Type":"ContainerStarted","Data":"ad9d3aafabbe4aa1dc486a5a428699eb337b5538745fc024cf99692e6949c7e3"} Jul 06 19:55:09.941858 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:09.941809 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" podStartSLOduration=1.941792027 podStartE2EDuration="1.941792027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:55:09.940867074 +0000 UTC m=+2606.836756607" watchObservedRunningTime="2026-07-06 19:55:09.941792027 +0000 UTC m=+2606.837681558" Jul 06 19:55:19.236766 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:19.236698 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:19.237209 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:19.236794 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:19.250621 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:19.250589 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:19.991528 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:19.991497 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:55:37.483610 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:37.483563 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:56.299169 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:56.299125 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:57.217195 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:57.217156 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r"] Jul 06 19:55:57.242280 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:55:57.242241 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:05.600625 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:05.600583 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:05.699946 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:05.699904 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:12.237578 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:12.237529 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" podUID="6dab534c-58d8-4a41-82f3-605234244dd3" containerName="main" containerID="cri-o://a9a22943214aa528cca7c399da01db0cb393a556178837f634849214cbe7337b" gracePeriod=45 Jul 06 19:56:12.498406 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:12.498332 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:56:12.559225 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:12.559192 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dab534c-58d8-4a41-82f3-605234244dd3-tls-certs\") pod \"6dab534c-58d8-4a41-82f3-605234244dd3\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " Jul 06 19:56:12.559432 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:12.559240 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-dshm\") pod \"6dab534c-58d8-4a41-82f3-605234244dd3\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " Jul 06 19:56:12.559432 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:12.559276 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t45ph\" (UniqueName: \"kubernetes.io/projected/6dab534c-58d8-4a41-82f3-605234244dd3-kube-api-access-t45ph\") pod \"6dab534c-58d8-4a41-82f3-605234244dd3\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " Jul 06 19:56:12.559432 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:12.559304 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-home\") pod \"6dab534c-58d8-4a41-82f3-605234244dd3\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " Jul 06 19:56:12.559432 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:12.559326 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-tmp-dir\") pod \"6dab534c-58d8-4a41-82f3-605234244dd3\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " Jul 06 19:56:12.559432 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:12.559344 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-model-cache\") pod \"6dab534c-58d8-4a41-82f3-605234244dd3\" (UID: \"6dab534c-58d8-4a41-82f3-605234244dd3\") " Jul 06 19:56:12.559711 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:12.559632 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6dab534c-58d8-4a41-82f3-605234244dd3" (UID: "6dab534c-58d8-4a41-82f3-605234244dd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:12.559711 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:12.559615 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-home" (OuterVolumeSpecName: "home") pod "6dab534c-58d8-4a41-82f3-605234244dd3" (UID: "6dab534c-58d8-4a41-82f3-605234244dd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:12.559823 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:12.559690 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-model-cache" (OuterVolumeSpecName: "model-cache") pod "6dab534c-58d8-4a41-82f3-605234244dd3" (UID: "6dab534c-58d8-4a41-82f3-605234244dd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:12.561517 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:12.561489 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-dshm" (OuterVolumeSpecName: "dshm") pod "6dab534c-58d8-4a41-82f3-605234244dd3" (UID: "6dab534c-58d8-4a41-82f3-605234244dd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:12.561598 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:12.561561 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dab534c-58d8-4a41-82f3-605234244dd3-kube-api-access-t45ph" (OuterVolumeSpecName: "kube-api-access-t45ph") pod "6dab534c-58d8-4a41-82f3-605234244dd3" (UID: "6dab534c-58d8-4a41-82f3-605234244dd3"). InnerVolumeSpecName "kube-api-access-t45ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:56:12.561652 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:12.561593 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dab534c-58d8-4a41-82f3-605234244dd3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6dab534c-58d8-4a41-82f3-605234244dd3" (UID: "6dab534c-58d8-4a41-82f3-605234244dd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:56:12.660513 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:12.660350 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-tmp-dir\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:56:12.660513 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:12.660517 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-model-cache\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:56:12.660807 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:12.660531 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dab534c-58d8-4a41-82f3-605234244dd3-tls-certs\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:56:12.660807 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:12.660542 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-dshm\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:56:12.660807 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:12.660557 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t45ph\" (UniqueName: \"kubernetes.io/projected/6dab534c-58d8-4a41-82f3-605234244dd3-kube-api-access-t45ph\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:56:12.660807 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:12.660568 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dab534c-58d8-4a41-82f3-605234244dd3-home\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:56:13.275716 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:13.275681 2566 generic.go:356] "Generic (PLEG): container finished" podID="6dab534c-58d8-4a41-82f3-605234244dd3" containerID="a9a22943214aa528cca7c399da01db0cb393a556178837f634849214cbe7337b" exitCode=0 Jul 06 19:56:13.276207 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:13.275754 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" Jul 06 19:56:13.276207 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:13.275779 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" event={"ID":"6dab534c-58d8-4a41-82f3-605234244dd3","Type":"ContainerDied","Data":"a9a22943214aa528cca7c399da01db0cb393a556178837f634849214cbe7337b"} Jul 06 19:56:13.276207 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:13.275827 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r" event={"ID":"6dab534c-58d8-4a41-82f3-605234244dd3","Type":"ContainerDied","Data":"ad9d3aafabbe4aa1dc486a5a428699eb337b5538745fc024cf99692e6949c7e3"} Jul 06 19:56:13.276207 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:13.275846 2566 scope.go:117] "RemoveContainer" containerID="a9a22943214aa528cca7c399da01db0cb393a556178837f634849214cbe7337b" Jul 06 19:56:13.285160 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:13.285143 2566 scope.go:117] "RemoveContainer" containerID="a9a22943214aa528cca7c399da01db0cb393a556178837f634849214cbe7337b" Jul 06 19:56:13.285404 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:56:13.285390 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9a22943214aa528cca7c399da01db0cb393a556178837f634849214cbe7337b\": container with ID starting with a9a22943214aa528cca7c399da01db0cb393a556178837f634849214cbe7337b not found: ID does not exist" containerID="a9a22943214aa528cca7c399da01db0cb393a556178837f634849214cbe7337b" Jul 06 19:56:13.285449 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:13.285413 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9a22943214aa528cca7c399da01db0cb393a556178837f634849214cbe7337b"} err="failed to get container status \"a9a22943214aa528cca7c399da01db0cb393a556178837f634849214cbe7337b\": rpc error: code = NotFound desc = could not find container \"a9a22943214aa528cca7c399da01db0cb393a556178837f634849214cbe7337b\": container with ID starting with a9a22943214aa528cca7c399da01db0cb393a556178837f634849214cbe7337b not found: ID does not exist" Jul 06 19:56:13.297532 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:13.297507 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r"] Jul 06 19:56:13.305297 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:13.305273 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c6207b/sub-test-model-c6207b-kserve-766f8fcc9-87z6r"] Jul 06 19:56:13.763451 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:13.763420 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dab534c-58d8-4a41-82f3-605234244dd3" path="/var/lib/kubelet/pods/6dab534c-58d8-4a41-82f3-605234244dd3/volumes" Jul 06 19:56:22.660177 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:22.660135 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:43.838024 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:43.837991 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:56:43.864855 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:43.864831 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2mdkh_a2d615bf-2066-4a10-b984-d7c2be1c8b6d/ovn-acl-logging/0.log" Jul 06 19:56:47.328435 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.328394 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f"] Jul 06 19:56:47.328821 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.328814 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dab534c-58d8-4a41-82f3-605234244dd3" containerName="main" Jul 06 19:56:47.328871 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.328826 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dab534c-58d8-4a41-82f3-605234244dd3" containerName="main" Jul 06 19:56:47.328907 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.328895 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dab534c-58d8-4a41-82f3-605234244dd3" containerName="main" Jul 06 19:56:47.333144 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.333110 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:47.339391 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.339362 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6207b\"/\"openshift-service-ca.crt\"" Jul 06 19:56:47.339391 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.339375 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6207b\"/\"default-dockercfg-jzxfb\"" Jul 06 19:56:47.339587 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.339373 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6207b\"/\"rate-test-model-c6207b-kserve-self-signed-certs\"" Jul 06 19:56:47.339587 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.339362 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6207b\"/\"kube-root-ca.crt\"" Jul 06 19:56:47.374165 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.374132 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f"] Jul 06 19:56:47.411125 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.411067 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:47.474895 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.474864 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-dshm\") pod \"rate-test-model-c6207b-kserve-778ccd9fff-l9b4f\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:47.475100 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.474914 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-home\") pod \"rate-test-model-c6207b-kserve-778ccd9fff-l9b4f\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:47.475100 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.474974 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-tmp-dir\") pod \"rate-test-model-c6207b-kserve-778ccd9fff-l9b4f\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:47.475100 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.475013 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-model-cache\") pod \"rate-test-model-c6207b-kserve-778ccd9fff-l9b4f\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:47.475100 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.475036 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75sqj\" (UniqueName: \"kubernetes.io/projected/c171b844-2493-4f59-a726-a0e94d5a7223-kube-api-access-75sqj\") pod \"rate-test-model-c6207b-kserve-778ccd9fff-l9b4f\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:47.475269 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.475131 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c171b844-2493-4f59-a726-a0e94d5a7223-tls-certs\") pod \"rate-test-model-c6207b-kserve-778ccd9fff-l9b4f\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:47.576103 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.576066 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-dshm\") pod \"rate-test-model-c6207b-kserve-778ccd9fff-l9b4f\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:47.576103 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.576114 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-home\") pod \"rate-test-model-c6207b-kserve-778ccd9fff-l9b4f\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:47.576374 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.576140 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-tmp-dir\") pod \"rate-test-model-c6207b-kserve-778ccd9fff-l9b4f\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:47.576374 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.576171 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-model-cache\") pod \"rate-test-model-c6207b-kserve-778ccd9fff-l9b4f\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:47.576374 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.576199 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75sqj\" (UniqueName: \"kubernetes.io/projected/c171b844-2493-4f59-a726-a0e94d5a7223-kube-api-access-75sqj\") pod \"rate-test-model-c6207b-kserve-778ccd9fff-l9b4f\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:47.576374 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.576274 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c171b844-2493-4f59-a726-a0e94d5a7223-tls-certs\") pod \"rate-test-model-c6207b-kserve-778ccd9fff-l9b4f\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:47.576576 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.576549 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-home\") pod \"rate-test-model-c6207b-kserve-778ccd9fff-l9b4f\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:47.576701 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.576645 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-model-cache\") pod \"rate-test-model-c6207b-kserve-778ccd9fff-l9b4f\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:47.576701 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.576605 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-tmp-dir\") pod \"rate-test-model-c6207b-kserve-778ccd9fff-l9b4f\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:47.578569 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.578514 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-dshm\") pod \"rate-test-model-c6207b-kserve-778ccd9fff-l9b4f\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:47.578804 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.578783 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c171b844-2493-4f59-a726-a0e94d5a7223-tls-certs\") pod \"rate-test-model-c6207b-kserve-778ccd9fff-l9b4f\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:47.585228 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.585204 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75sqj\" (UniqueName: \"kubernetes.io/projected/c171b844-2493-4f59-a726-a0e94d5a7223-kube-api-access-75sqj\") pod \"rate-test-model-c6207b-kserve-778ccd9fff-l9b4f\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:47.642144 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.642105 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:47.789880 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:47.789857 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f"] Jul 06 19:56:47.791734 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:56:47.791686 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc171b844_2493_4f59_a726_a0e94d5a7223.slice/crio-617096158b9f1f1d4c32bd7bcf902cfeeb47eaf6c3c14dcf9270874ba0b03a23 WatchSource:0}: Error finding container 617096158b9f1f1d4c32bd7bcf902cfeeb47eaf6c3c14dcf9270874ba0b03a23: Status 404 returned error can't find the container with id 617096158b9f1f1d4c32bd7bcf902cfeeb47eaf6c3c14dcf9270874ba0b03a23 Jul 06 19:56:48.474118 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:48.474080 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" event={"ID":"c171b844-2493-4f59-a726-a0e94d5a7223","Type":"ContainerStarted","Data":"05b11d6f6a18b93e219f4422221cc9b9071b337a670865bd0bb34d6cc5df5b09"} Jul 06 19:56:48.474118 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:48.474126 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" event={"ID":"c171b844-2493-4f59-a726-a0e94d5a7223","Type":"ContainerStarted","Data":"617096158b9f1f1d4c32bd7bcf902cfeeb47eaf6c3c14dcf9270874ba0b03a23"} Jul 06 19:56:57.643176 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:57.643137 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:57.643699 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:57.643204 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:57.655610 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:57.655588 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:56:57.689912 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:57.689864 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" podStartSLOduration=10.689847793 podStartE2EDuration="10.689847793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:56:48.537404347 +0000 UTC m=+2705.433293880" watchObservedRunningTime="2026-07-06 19:56:57.689847793 +0000 UTC m=+2714.585737325" Jul 06 19:56:58.543394 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:56:58.543364 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:57:12.132241 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:12.132196 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:12.232503 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:12.232459 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:21.384747 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:21.384694 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:21.486471 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:21.486439 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:37.958428 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:37.958344 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:38.721220 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:38.721176 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f"] Jul 06 19:57:38.745687 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:38.745652 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:39.869574 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:39.869543 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:40.241110 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:40.241079 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:40.362610 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:40.362582 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:40.491997 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:40.491907 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:40.777692 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:40.777600 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:45.325535 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:45.325501 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:53.741603 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:53.741563 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" podUID="c171b844-2493-4f59-a726-a0e94d5a7223" containerName="main" containerID="cri-o://05b11d6f6a18b93e219f4422221cc9b9071b337a670865bd0bb34d6cc5df5b09" gracePeriod=45 Jul 06 19:57:54.000919 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.000853 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:57:54.090153 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.090118 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-tmp-dir\") pod \"c171b844-2493-4f59-a726-a0e94d5a7223\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " Jul 06 19:57:54.090349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.090172 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75sqj\" (UniqueName: \"kubernetes.io/projected/c171b844-2493-4f59-a726-a0e94d5a7223-kube-api-access-75sqj\") pod \"c171b844-2493-4f59-a726-a0e94d5a7223\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " Jul 06 19:57:54.090349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.090195 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c171b844-2493-4f59-a726-a0e94d5a7223-tls-certs\") pod \"c171b844-2493-4f59-a726-a0e94d5a7223\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " Jul 06 19:57:54.090349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.090220 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-dshm\") pod \"c171b844-2493-4f59-a726-a0e94d5a7223\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " Jul 06 19:57:54.090349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.090253 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-home\") pod \"c171b844-2493-4f59-a726-a0e94d5a7223\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " Jul 06 19:57:54.090349 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.090301 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-model-cache\") pod \"c171b844-2493-4f59-a726-a0e94d5a7223\" (UID: \"c171b844-2493-4f59-a726-a0e94d5a7223\") " Jul 06 19:57:54.090617 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.090399 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c171b844-2493-4f59-a726-a0e94d5a7223" (UID: "c171b844-2493-4f59-a726-a0e94d5a7223"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:57:54.090617 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.090587 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-home" (OuterVolumeSpecName: "home") pod "c171b844-2493-4f59-a726-a0e94d5a7223" (UID: "c171b844-2493-4f59-a726-a0e94d5a7223"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:57:54.090617 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.090603 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-model-cache" (OuterVolumeSpecName: "model-cache") pod "c171b844-2493-4f59-a726-a0e94d5a7223" (UID: "c171b844-2493-4f59-a726-a0e94d5a7223"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:57:54.090759 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.090672 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-tmp-dir\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:57:54.092391 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.092365 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-dshm" (OuterVolumeSpecName: "dshm") pod "c171b844-2493-4f59-a726-a0e94d5a7223" (UID: "c171b844-2493-4f59-a726-a0e94d5a7223"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:57:54.092391 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.092384 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c171b844-2493-4f59-a726-a0e94d5a7223-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c171b844-2493-4f59-a726-a0e94d5a7223" (UID: "c171b844-2493-4f59-a726-a0e94d5a7223"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:57:54.092539 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.092381 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c171b844-2493-4f59-a726-a0e94d5a7223-kube-api-access-75sqj" (OuterVolumeSpecName: "kube-api-access-75sqj") pod "c171b844-2493-4f59-a726-a0e94d5a7223" (UID: "c171b844-2493-4f59-a726-a0e94d5a7223"). InnerVolumeSpecName "kube-api-access-75sqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:57:54.191214 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.191176 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-model-cache\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:57:54.191214 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.191214 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75sqj\" (UniqueName: \"kubernetes.io/projected/c171b844-2493-4f59-a726-a0e94d5a7223-kube-api-access-75sqj\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:57:54.191214 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.191226 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c171b844-2493-4f59-a726-a0e94d5a7223-tls-certs\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:57:54.191463 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.191234 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-dshm\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:57:54.191463 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.191242 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c171b844-2493-4f59-a726-a0e94d5a7223-home\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:57:54.843702 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.843667 2566 generic.go:356] "Generic (PLEG): container finished" podID="c171b844-2493-4f59-a726-a0e94d5a7223" containerID="05b11d6f6a18b93e219f4422221cc9b9071b337a670865bd0bb34d6cc5df5b09" exitCode=0 Jul 06 19:57:54.844198 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.843746 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" Jul 06 19:57:54.844198 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.843757 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" event={"ID":"c171b844-2493-4f59-a726-a0e94d5a7223","Type":"ContainerDied","Data":"05b11d6f6a18b93e219f4422221cc9b9071b337a670865bd0bb34d6cc5df5b09"} Jul 06 19:57:54.844198 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.843804 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f" event={"ID":"c171b844-2493-4f59-a726-a0e94d5a7223","Type":"ContainerDied","Data":"617096158b9f1f1d4c32bd7bcf902cfeeb47eaf6c3c14dcf9270874ba0b03a23"} Jul 06 19:57:54.844198 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.843828 2566 scope.go:117] "RemoveContainer" containerID="05b11d6f6a18b93e219f4422221cc9b9071b337a670865bd0bb34d6cc5df5b09" Jul 06 19:57:54.853374 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.853355 2566 scope.go:117] "RemoveContainer" containerID="05b11d6f6a18b93e219f4422221cc9b9071b337a670865bd0bb34d6cc5df5b09" Jul 06 19:57:54.853666 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:57:54.853644 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b11d6f6a18b93e219f4422221cc9b9071b337a670865bd0bb34d6cc5df5b09\": container with ID starting with 05b11d6f6a18b93e219f4422221cc9b9071b337a670865bd0bb34d6cc5df5b09 not found: ID does not exist" containerID="05b11d6f6a18b93e219f4422221cc9b9071b337a670865bd0bb34d6cc5df5b09" Jul 06 19:57:54.853766 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.853677 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b11d6f6a18b93e219f4422221cc9b9071b337a670865bd0bb34d6cc5df5b09"} err="failed to get container status \"05b11d6f6a18b93e219f4422221cc9b9071b337a670865bd0bb34d6cc5df5b09\": rpc error: code = NotFound desc = could not find container \"05b11d6f6a18b93e219f4422221cc9b9071b337a670865bd0bb34d6cc5df5b09\": container with ID starting with 05b11d6f6a18b93e219f4422221cc9b9071b337a670865bd0bb34d6cc5df5b09 not found: ID does not exist" Jul 06 19:57:54.867962 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.867930 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f"] Jul 06 19:57:54.873729 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:54.873689 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c6207b/rate-test-model-c6207b-kserve-778ccd9fff-l9b4f"] Jul 06 19:57:55.764083 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:57:55.764052 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c171b844-2493-4f59-a726-a0e94d5a7223" path="/var/lib/kubelet/pods/c171b844-2493-4f59-a726-a0e94d5a7223/volumes" Jul 06 19:58:10.491734 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:10.491633 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:22.317249 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:22.317210 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:22.604933 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:22.604840 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:22.847215 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:22.847182 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:23.023894 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:23.023860 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:23.479013 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:23.478983 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:23.578988 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:23.578948 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:23.678993 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:23.678960 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:23.780012 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:23.779879 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:28.583678 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:28.583644 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:29.924463 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:29.924434 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:30.039794 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:30.039761 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:30.283141 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:30.283049 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj"] Jul 06 19:58:30.283321 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:30.283279 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" podUID="1c41ef09-8de4-478c-bf7f-af9a1292bb2d" containerName="istio-proxy" containerID="cri-o://251696666b3fdbc45f6276b374a69f4d1e009cdaf3b9181ae15eb737f0bae9e0" gracePeriod=30 Jul 06 19:58:30.299178 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:30.299148 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:30.509110 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:30.509077 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:30.920966 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:30.920931 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58b889c56f-smm9k_a0c1539f-c35c-4a6d-8b6c-288e7b0a9ecb/authorino/0.log" Jul 06 19:58:35.552377 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.552310 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:58:35.683544 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.683498 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-token\") pod \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " Jul 06 19:58:35.683544 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.683553 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6plz4\" (UniqueName: \"kubernetes.io/projected/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-kube-api-access-6plz4\") pod \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " Jul 06 19:58:35.683871 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.683592 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-credential-socket\") pod \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " Jul 06 19:58:35.683871 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.683610 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-workload-socket\") pod \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " Jul 06 19:58:35.683871 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.683639 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istiod-ca-cert\") pod \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " Jul 06 19:58:35.683871 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.683671 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-data\") pod \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " Jul 06 19:58:35.683871 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.683823 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-envoy\") pod \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " Jul 06 19:58:35.683871 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.683848 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-workload-certs\") pod \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " Jul 06 19:58:35.684175 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.683933 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-podinfo\") pod \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\" (UID: \"1c41ef09-8de4-478c-bf7f-af9a1292bb2d\") " Jul 06 19:58:35.684175 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.683981 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1c41ef09-8de4-478c-bf7f-af9a1292bb2d" (UID: "1c41ef09-8de4-478c-bf7f-af9a1292bb2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:58:35.684175 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.684049 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1c41ef09-8de4-478c-bf7f-af9a1292bb2d" (UID: "1c41ef09-8de4-478c-bf7f-af9a1292bb2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:58:35.684328 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.684227 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1c41ef09-8de4-478c-bf7f-af9a1292bb2d" (UID: "1c41ef09-8de4-478c-bf7f-af9a1292bb2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:58:35.684328 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.684258 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1c41ef09-8de4-478c-bf7f-af9a1292bb2d" (UID: "1c41ef09-8de4-478c-bf7f-af9a1292bb2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:58:35.684545 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.684528 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istiod-ca-cert\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:58:35.684630 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.684549 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-workload-certs\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:58:35.684630 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.684562 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-credential-socket\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:58:35.684630 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.684575 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-workload-socket\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:58:35.684885 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.684859 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-data" (OuterVolumeSpecName: "istio-data") pod "1c41ef09-8de4-478c-bf7f-af9a1292bb2d" (UID: "1c41ef09-8de4-478c-bf7f-af9a1292bb2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:58:35.686021 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.685987 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1c41ef09-8de4-478c-bf7f-af9a1292bb2d" (UID: "1c41ef09-8de4-478c-bf7f-af9a1292bb2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:58:35.686120 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.686034 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-token" (OuterVolumeSpecName: "istio-token") pod "1c41ef09-8de4-478c-bf7f-af9a1292bb2d" (UID: "1c41ef09-8de4-478c-bf7f-af9a1292bb2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:58:35.686120 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.686067 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-kube-api-access-6plz4" (OuterVolumeSpecName: "kube-api-access-6plz4") pod "1c41ef09-8de4-478c-bf7f-af9a1292bb2d" (UID: "1c41ef09-8de4-478c-bf7f-af9a1292bb2d"). InnerVolumeSpecName "kube-api-access-6plz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:58:35.686195 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.686119 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1c41ef09-8de4-478c-bf7f-af9a1292bb2d" (UID: "1c41ef09-8de4-478c-bf7f-af9a1292bb2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:58:35.785876 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.785847 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6plz4\" (UniqueName: \"kubernetes.io/projected/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-kube-api-access-6plz4\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:58:35.785876 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.785874 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-data\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:58:35.786038 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.785883 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-envoy\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:58:35.786038 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.785892 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-podinfo\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:58:35.786038 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.785900 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c41ef09-8de4-478c-bf7f-af9a1292bb2d-istio-token\") on node \"ip-10-0-133-201.ec2.internal\" DevicePath \"\"" Jul 06 19:58:35.838231 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.838130 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xwb8h_5141d320-7665-460c-b7c4-f9e2202470cf/manager/0.log" Jul 06 19:58:35.983431 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:35.983400 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-cq9gn_2537d88a-bc4c-46a2-9f91-9620fe11fc1a/manager/0.log" Jul 06 19:58:36.075157 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:36.075116 2566 generic.go:356] "Generic (PLEG): container finished" podID="1c41ef09-8de4-478c-bf7f-af9a1292bb2d" containerID="251696666b3fdbc45f6276b374a69f4d1e009cdaf3b9181ae15eb737f0bae9e0" exitCode=0 Jul 06 19:58:36.075331 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:36.075163 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" Jul 06 19:58:36.075331 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:36.075208 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" event={"ID":"1c41ef09-8de4-478c-bf7f-af9a1292bb2d","Type":"ContainerDied","Data":"251696666b3fdbc45f6276b374a69f4d1e009cdaf3b9181ae15eb737f0bae9e0"} Jul 06 19:58:36.075331 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:36.075260 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj" event={"ID":"1c41ef09-8de4-478c-bf7f-af9a1292bb2d","Type":"ContainerDied","Data":"8812f7894a788412a0b430302f6653c3258da59b708dcc01ad4ca764ccee6a67"} Jul 06 19:58:36.075331 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:36.075286 2566 scope.go:117] "RemoveContainer" containerID="251696666b3fdbc45f6276b374a69f4d1e009cdaf3b9181ae15eb737f0bae9e0" Jul 06 19:58:36.084733 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:36.084698 2566 scope.go:117] "RemoveContainer" containerID="f687a08847b295cce0fdf8758772692735523fa0c0c98d0db37636a1e24c835c" Jul 06 19:58:36.092951 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:36.092929 2566 scope.go:117] "RemoveContainer" containerID="251696666b3fdbc45f6276b374a69f4d1e009cdaf3b9181ae15eb737f0bae9e0" Jul 06 19:58:36.093224 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:58:36.093205 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"251696666b3fdbc45f6276b374a69f4d1e009cdaf3b9181ae15eb737f0bae9e0\": container with ID starting with 251696666b3fdbc45f6276b374a69f4d1e009cdaf3b9181ae15eb737f0bae9e0 not found: ID does not exist" containerID="251696666b3fdbc45f6276b374a69f4d1e009cdaf3b9181ae15eb737f0bae9e0" Jul 06 19:58:36.093276 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:36.093234 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"251696666b3fdbc45f6276b374a69f4d1e009cdaf3b9181ae15eb737f0bae9e0"} err="failed to get container status \"251696666b3fdbc45f6276b374a69f4d1e009cdaf3b9181ae15eb737f0bae9e0\": rpc error: code = NotFound desc = could not find container \"251696666b3fdbc45f6276b374a69f4d1e009cdaf3b9181ae15eb737f0bae9e0\": container with ID starting with 251696666b3fdbc45f6276b374a69f4d1e009cdaf3b9181ae15eb737f0bae9e0 not found: ID does not exist" Jul 06 19:58:36.093276 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:36.093256 2566 scope.go:117] "RemoveContainer" containerID="f687a08847b295cce0fdf8758772692735523fa0c0c98d0db37636a1e24c835c" Jul 06 19:58:36.093505 ip-10-0-133-201 kubenswrapper[2566]: E0706 19:58:36.093485 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f687a08847b295cce0fdf8758772692735523fa0c0c98d0db37636a1e24c835c\": container with ID starting with f687a08847b295cce0fdf8758772692735523fa0c0c98d0db37636a1e24c835c not found: ID does not exist" containerID="f687a08847b295cce0fdf8758772692735523fa0c0c98d0db37636a1e24c835c" Jul 06 19:58:36.093576 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:36.093508 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f687a08847b295cce0fdf8758772692735523fa0c0c98d0db37636a1e24c835c"} err="failed to get container status \"f687a08847b295cce0fdf8758772692735523fa0c0c98d0db37636a1e24c835c\": rpc error: code = NotFound desc = could not find container \"f687a08847b295cce0fdf8758772692735523fa0c0c98d0db37636a1e24c835c\": container with ID starting with f687a08847b295cce0fdf8758772692735523fa0c0c98d0db37636a1e24c835c not found: ID does not exist" Jul 06 19:58:36.096689 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:36.096666 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj"] Jul 06 19:58:36.101784 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:36.101760 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-c6207b-openshift-default-7c54d67999-gmgbj"] Jul 06 19:58:36.488985 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:36.488947 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c475d6674-wlhcj_223aa52f-6613-49da-bec7-dd08654cb587/manager/0.log" Jul 06 19:58:36.626146 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:36.626121 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-d2scm_0e5da542-3f91-4f05-8959-c4c05f9210a9/server/0.log" Jul 06 19:58:36.744420 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:36.744337 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lf8hn_33a0983c-34cc-4244-99d0-7e3e69f94bbc/manager/1.log" Jul 06 19:58:37.142191 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:37.142099 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56f978bf57-w7bsg_4ca071df-c56c-4c75-bd57-f30e0b31976f/copy-manifests/0.log" Jul 06 19:58:37.148838 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:37.148795 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56f978bf57-w7bsg_4ca071df-c56c-4c75-bd57-f30e0b31976f/manager/0.log" Jul 06 19:58:37.266101 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:37.266067 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-5j9s4_f53cbc16-d0a4-4020-810f-2a09d6a2755b/postgres/0.log" Jul 06 19:58:37.764682 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:37.764649 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c41ef09-8de4-478c-bf7f-af9a1292bb2d" path="/var/lib/kubelet/pods/1c41ef09-8de4-478c-bf7f-af9a1292bb2d/volumes" Jul 06 19:58:38.090555 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:38.090476 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm_da1a9d25-a8bf-4d2b-9531-c15710aa8192/util/0.log" Jul 06 19:58:38.097832 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:38.097805 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm_da1a9d25-a8bf-4d2b-9531-c15710aa8192/pull/0.log" Jul 06 19:58:38.105420 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:38.105389 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm_da1a9d25-a8bf-4d2b-9531-c15710aa8192/extract/0.log" Jul 06 19:58:38.220383 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:38.220354 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85_90fd6204-edb5-4cce-aee2-c576f74414e2/util/0.log" Jul 06 19:58:38.227252 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:38.227225 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85_90fd6204-edb5-4cce-aee2-c576f74414e2/pull/0.log" Jul 06 19:58:38.233842 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:38.233822 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85_90fd6204-edb5-4cce-aee2-c576f74414e2/extract/0.log" Jul 06 19:58:38.356510 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:38.356433 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4_a62f42d5-f4f7-48bf-ad8b-e87e673bac72/util/0.log" Jul 06 19:58:38.363478 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:38.363455 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4_a62f42d5-f4f7-48bf-ad8b-e87e673bac72/pull/0.log" Jul 06 19:58:38.371195 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:38.371170 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4_a62f42d5-f4f7-48bf-ad8b-e87e673bac72/extract/0.log" Jul 06 19:58:38.502161 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:38.502133 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn_ab2c8e51-541d-4f44-9d87-a64dd000f2d7/util/0.log" Jul 06 19:58:38.513403 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:38.513375 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn_ab2c8e51-541d-4f44-9d87-a64dd000f2d7/pull/0.log" Jul 06 19:58:38.522636 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:38.522616 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn_ab2c8e51-541d-4f44-9d87-a64dd000f2d7/extract/0.log" Jul 06 19:58:38.646085 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:38.645979 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58b889c56f-smm9k_a0c1539f-c35c-4a6d-8b6c-288e7b0a9ecb/authorino/0.log" Jul 06 19:58:38.766439 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:38.766402 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-v46zz_52062462-8aad-4dc8-9d1c-bae22f5c4904/manager/0.log" Jul 06 19:58:38.883648 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:38.883623 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-zt5cm_9e96fdac-3e0c-43ba-8ed0-d94881436b66/manager/0.log" Jul 06 19:58:39.137676 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:39.137651 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-svpfq_91d7dfa4-0892-4cac-bbeb-34565f1c8540/registry-server/0.log" Jul 06 19:58:39.525425 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:39.525399 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-ccj7m_c3b22f65-0d6a-4bd6-82e9-32630f1411b5/manager/0.log" Jul 06 19:58:39.899964 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:39.899878 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4rzbhr_a5e16a95-6efc-4765-9e90-1a5f17af02e3/istio-proxy/0.log" Jul 06 19:58:40.150952 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:40.150864 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fffdf4997-n76hp_b354c43c-54c3-4679-9bc3-dea447406712/kube-auth-proxy/0.log" Jul 06 19:58:40.414262 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:40.414228 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-f9c2h_91828ea8-792a-4816-a022-1b6bb2e8c19c/istio-proxy/1.log" Jul 06 19:58:40.537113 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:40.537070 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-z4dzj_11b1cacc-17e2-43f2-874b-febb142224bb/payload-pre-processing/0.log" Jul 06 19:58:40.673247 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:40.673153 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-fhccx_54d38cb0-b022-4644-a5d4-7e8e8d908019/payload-processing/0.log" Jul 06 19:58:41.281372 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:41.281345 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-jhjvw_ab9c5d7e-99ec-4324-be07-9d7d00898af7/main/0.log" Jul 06 19:58:41.528509 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:41.528470 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs47hp_b8d81d6c-9c20-4284-86c1-7a9a26c1a6d2/main/0.log" Jul 06 19:58:45.888121 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:45.888088 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:45.986166 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:45.986133 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:49.144607 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:49.144562 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qrqcb_4240cab6-4ae1-4d08-ae5d-d3040f4f9d61/global-pull-secret-syncer/0.log" Jul 06 19:58:49.270110 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:49.270080 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2cnx7_b0dba382-5dc7-4c19-bad4-46d9ad717afc/konnectivity-agent/0.log" Jul 06 19:58:49.352206 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:49.352176 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-201.ec2.internal_a35ae4d1cb32eff9744780fd38da560f/haproxy/0.log" Jul 06 19:58:53.675932 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:53.675882 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm_da1a9d25-a8bf-4d2b-9531-c15710aa8192/extract/0.log" Jul 06 19:58:53.700962 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:53.700934 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm_da1a9d25-a8bf-4d2b-9531-c15710aa8192/util/0.log" Jul 06 19:58:53.725674 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:53.725646 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75942ddm_da1a9d25-a8bf-4d2b-9531-c15710aa8192/pull/0.log" Jul 06 19:58:53.754830 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:53.754789 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85_90fd6204-edb5-4cce-aee2-c576f74414e2/extract/0.log" Jul 06 19:58:53.779591 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:53.779564 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85_90fd6204-edb5-4cce-aee2-c576f74414e2/util/0.log" Jul 06 19:58:53.814118 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:53.814091 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jfl85_90fd6204-edb5-4cce-aee2-c576f74414e2/pull/0.log" Jul 06 19:58:53.847034 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:53.847005 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4_a62f42d5-f4f7-48bf-ad8b-e87e673bac72/extract/0.log" Jul 06 19:58:53.872052 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:53.872021 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4_a62f42d5-f4f7-48bf-ad8b-e87e673bac72/util/0.log" Jul 06 19:58:53.896560 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:53.896527 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zc2r4_a62f42d5-f4f7-48bf-ad8b-e87e673bac72/pull/0.log" Jul 06 19:58:53.927864 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:53.927792 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn_ab2c8e51-541d-4f44-9d87-a64dd000f2d7/extract/0.log" Jul 06 19:58:53.952098 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:53.952073 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn_ab2c8e51-541d-4f44-9d87-a64dd000f2d7/util/0.log" Jul 06 19:58:53.975328 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:53.975302 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12kmfn_ab2c8e51-541d-4f44-9d87-a64dd000f2d7/pull/0.log" Jul 06 19:58:54.244819 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:54.244706 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58b889c56f-smm9k_a0c1539f-c35c-4a6d-8b6c-288e7b0a9ecb/authorino/0.log" Jul 06 19:58:54.312174 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:54.312147 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-v46zz_52062462-8aad-4dc8-9d1c-bae22f5c4904/manager/0.log" Jul 06 19:58:54.344599 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:54.344567 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-zt5cm_9e96fdac-3e0c-43ba-8ed0-d94881436b66/manager/0.log" Jul 06 19:58:54.424774 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:54.424745 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-svpfq_91d7dfa4-0892-4cac-bbeb-34565f1c8540/registry-server/0.log" Jul 06 19:58:54.748218 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:54.748184 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-ccj7m_c3b22f65-0d6a-4bd6-82e9-32630f1411b5/manager/0.log" Jul 06 19:58:56.592117 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:56.592082 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-9rhvm_ce7fbf1a-70ea-478d-914f-ca46298998aa/kube-state-metrics/0.log" Jul 06 19:58:56.611990 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:56.611964 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-9rhvm_ce7fbf1a-70ea-478d-914f-ca46298998aa/kube-rbac-proxy-main/0.log" Jul 06 19:58:56.636416 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:56.636386 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-9rhvm_ce7fbf1a-70ea-478d-914f-ca46298998aa/kube-rbac-proxy-self/0.log" Jul 06 19:58:56.691679 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:56.691647 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-9ghq4_9b43d782-fd1c-4822-9944-dca222c4887a/monitoring-plugin/0.log" Jul 06 19:58:56.726740 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:56.726701 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2cpjz_f96f9815-cbfa-4b08-b3e9-52b7368232c6/node-exporter/0.log" Jul 06 19:58:56.750748 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:56.750710 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2cpjz_f96f9815-cbfa-4b08-b3e9-52b7368232c6/kube-rbac-proxy/0.log" Jul 06 19:58:56.774392 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:56.774365 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2cpjz_f96f9815-cbfa-4b08-b3e9-52b7368232c6/init-textfile/0.log" Jul 06 19:58:59.099695 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.099662 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng"] Jul 06 19:58:59.100096 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.100068 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c41ef09-8de4-478c-bf7f-af9a1292bb2d" containerName="istio-proxy" Jul 06 19:58:59.100096 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.100082 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c41ef09-8de4-478c-bf7f-af9a1292bb2d" containerName="istio-proxy" Jul 06 19:58:59.100171 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.100102 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c171b844-2493-4f59-a726-a0e94d5a7223" containerName="main" Jul 06 19:58:59.100171 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.100108 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c171b844-2493-4f59-a726-a0e94d5a7223" containerName="main" Jul 06 19:58:59.100171 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.100167 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c41ef09-8de4-478c-bf7f-af9a1292bb2d" containerName="istio-proxy" Jul 06 19:58:59.100297 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.100175 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c171b844-2493-4f59-a726-a0e94d5a7223" containerName="main" Jul 06 19:58:59.103425 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.103403 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" Jul 06 19:58:59.105954 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.105926 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bq5kf\"/\"kube-root-ca.crt\"" Jul 06 19:58:59.106078 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.106026 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bq5kf\"/\"default-dockercfg-xrlgm\"" Jul 06 19:58:59.106861 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.106841 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bq5kf\"/\"openshift-service-ca.crt\"" Jul 06 19:58:59.113361 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.113339 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng"] Jul 06 19:58:59.187042 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.186993 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f62s\" (UniqueName: \"kubernetes.io/projected/5dbca33f-562b-4888-bbb1-b42124271748-kube-api-access-5f62s\") pod \"perf-node-gather-daemonset-mbzng\" (UID: \"5dbca33f-562b-4888-bbb1-b42124271748\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" Jul 06 19:58:59.187042 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.187038 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5dbca33f-562b-4888-bbb1-b42124271748-proc\") pod \"perf-node-gather-daemonset-mbzng\" (UID: \"5dbca33f-562b-4888-bbb1-b42124271748\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" Jul 06 19:58:59.187290 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.187059 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5dbca33f-562b-4888-bbb1-b42124271748-podres\") pod \"perf-node-gather-daemonset-mbzng\" (UID: \"5dbca33f-562b-4888-bbb1-b42124271748\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" Jul 06 19:58:59.187290 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.187096 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5dbca33f-562b-4888-bbb1-b42124271748-lib-modules\") pod \"perf-node-gather-daemonset-mbzng\" (UID: \"5dbca33f-562b-4888-bbb1-b42124271748\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" Jul 06 19:58:59.187290 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.187122 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5dbca33f-562b-4888-bbb1-b42124271748-sys\") pod \"perf-node-gather-daemonset-mbzng\" (UID: \"5dbca33f-562b-4888-bbb1-b42124271748\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" Jul 06 19:58:59.288224 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.288171 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f62s\" (UniqueName: \"kubernetes.io/projected/5dbca33f-562b-4888-bbb1-b42124271748-kube-api-access-5f62s\") pod \"perf-node-gather-daemonset-mbzng\" (UID: \"5dbca33f-562b-4888-bbb1-b42124271748\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" Jul 06 19:58:59.288224 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.288230 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5dbca33f-562b-4888-bbb1-b42124271748-proc\") pod \"perf-node-gather-daemonset-mbzng\" (UID: \"5dbca33f-562b-4888-bbb1-b42124271748\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" Jul 06 19:58:59.288465 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.288250 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5dbca33f-562b-4888-bbb1-b42124271748-podres\") pod \"perf-node-gather-daemonset-mbzng\" (UID: \"5dbca33f-562b-4888-bbb1-b42124271748\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" Jul 06 19:58:59.288465 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.288276 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5dbca33f-562b-4888-bbb1-b42124271748-lib-modules\") pod \"perf-node-gather-daemonset-mbzng\" (UID: \"5dbca33f-562b-4888-bbb1-b42124271748\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" Jul 06 19:58:59.288465 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.288315 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5dbca33f-562b-4888-bbb1-b42124271748-sys\") pod \"perf-node-gather-daemonset-mbzng\" (UID: \"5dbca33f-562b-4888-bbb1-b42124271748\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" Jul 06 19:58:59.288465 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.288335 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5dbca33f-562b-4888-bbb1-b42124271748-proc\") pod \"perf-node-gather-daemonset-mbzng\" (UID: \"5dbca33f-562b-4888-bbb1-b42124271748\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" Jul 06 19:58:59.288465 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.288396 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5dbca33f-562b-4888-bbb1-b42124271748-podres\") pod \"perf-node-gather-daemonset-mbzng\" (UID: \"5dbca33f-562b-4888-bbb1-b42124271748\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" Jul 06 19:58:59.288465 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.288414 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5dbca33f-562b-4888-bbb1-b42124271748-sys\") pod \"perf-node-gather-daemonset-mbzng\" (UID: \"5dbca33f-562b-4888-bbb1-b42124271748\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" Jul 06 19:58:59.288671 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.288469 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5dbca33f-562b-4888-bbb1-b42124271748-lib-modules\") pod \"perf-node-gather-daemonset-mbzng\" (UID: \"5dbca33f-562b-4888-bbb1-b42124271748\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" Jul 06 19:58:59.296372 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.296350 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f62s\" (UniqueName: \"kubernetes.io/projected/5dbca33f-562b-4888-bbb1-b42124271748-kube-api-access-5f62s\") pod \"perf-node-gather-daemonset-mbzng\" (UID: \"5dbca33f-562b-4888-bbb1-b42124271748\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" Jul 06 19:58:59.415219 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.415191 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" Jul 06 19:58:59.553328 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:58:59.553302 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng"] Jul 06 19:58:59.555479 ip-10-0-133-201 kubenswrapper[2566]: W0706 19:58:59.555453 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5dbca33f_562b_4888_bbb1_b42124271748.slice/crio-80d591204a716225e370b10d6aab0555d38b8fe211fe423c721f523d8c3021d5 WatchSource:0}: Error finding container 80d591204a716225e370b10d6aab0555d38b8fe211fe423c721f523d8c3021d5: Status 404 returned error can't find the container with id 80d591204a716225e370b10d6aab0555d38b8fe211fe423c721f523d8c3021d5 Jul 06 19:59:00.212313 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:00.212274 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" event={"ID":"5dbca33f-562b-4888-bbb1-b42124271748","Type":"ContainerStarted","Data":"fa24b0301e282e5d6c442f5ade9a853d5808f3745bbe902f81dbee27ec7fa8c3"} Jul 06 19:59:00.212701 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:00.212330 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" event={"ID":"5dbca33f-562b-4888-bbb1-b42124271748","Type":"ContainerStarted","Data":"80d591204a716225e370b10d6aab0555d38b8fe211fe423c721f523d8c3021d5"} Jul 06 19:59:00.212701 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:00.212377 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" Jul 06 19:59:00.231512 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:00.231456 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" podStartSLOduration=1.231439376 podStartE2EDuration="1.231439376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:59:00.229363369 +0000 UTC m=+2837.125252902" watchObservedRunningTime="2026-07-06 19:59:00.231439376 +0000 UTC m=+2837.127328908" Jul 06 19:59:01.821416 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:01.821375 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-62xmt_bca4b412-168b-48a3-8251-28aea09c3427/dns/0.log" Jul 06 19:59:01.848220 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:01.848194 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-62xmt_bca4b412-168b-48a3-8251-28aea09c3427/kube-rbac-proxy/0.log" Jul 06 19:59:01.991490 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:01.991459 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zcnp6_a751eaf8-99c8-4e12-92a7-e135a18167a9/dns-node-resolver/0.log" Jul 06 19:59:02.647163 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:02.647135 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-454zs_5f592f36-c320-4ddb-abca-8bc3f6586ae0/node-ca/0.log" Jul 06 19:59:03.629800 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:03.629771 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4rzbhr_a5e16a95-6efc-4765-9e90-1a5f17af02e3/istio-proxy/0.log" Jul 06 19:59:03.801877 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:03.801852 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fffdf4997-n76hp_b354c43c-54c3-4679-9bc3-dea447406712/kube-auth-proxy/0.log" Jul 06 19:59:03.861192 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:03.861160 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-f9c2h_91828ea8-792a-4816-a022-1b6bb2e8c19c/istio-proxy/0.log" Jul 06 19:59:03.889284 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:03.889199 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-f9c2h_91828ea8-792a-4816-a022-1b6bb2e8c19c/istio-proxy/1.log" Jul 06 19:59:03.922360 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:03.922329 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-z4dzj_11b1cacc-17e2-43f2-874b-febb142224bb/payload-pre-processing/0.log" Jul 06 19:59:03.969979 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:03.969942 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-fhccx_54d38cb0-b022-4644-a5d4-7e8e8d908019/payload-processing/0.log" Jul 06 19:59:04.694415 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:04.694384 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mk4bf_3831b65a-0a72-4c06-b6d4-b3bff61a5a32/serve-healthcheck-canary/0.log" Jul 06 19:59:05.270257 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:05.270229 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-78fpq_82cca416-5961-4dae-ac5b-87f05514dea2/kube-rbac-proxy/0.log" Jul 06 19:59:05.293874 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:05.293850 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-78fpq_82cca416-5961-4dae-ac5b-87f05514dea2/exporter/0.log" Jul 06 19:59:05.319106 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:05.319078 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-78fpq_82cca416-5961-4dae-ac5b-87f05514dea2/extractor/0.log" Jul 06 19:59:06.227764 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:06.227735 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-mbzng" Jul 06 19:59:07.867109 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:07.867069 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xwb8h_5141d320-7665-460c-b7c4-f9e2202470cf/manager/0.log" Jul 06 19:59:08.002542 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:08.002457 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-cq9gn_2537d88a-bc4c-46a2-9f91-9620fe11fc1a/manager/0.log" Jul 06 19:59:08.205028 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:08.204999 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c475d6674-wlhcj_223aa52f-6613-49da-bec7-dd08654cb587/manager/0.log" Jul 06 19:59:08.229446 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:08.229422 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-d2scm_0e5da542-3f91-4f05-8959-c4c05f9210a9/server/0.log" Jul 06 19:59:08.293303 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:08.293213 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lf8hn_33a0983c-34cc-4244-99d0-7e3e69f94bbc/manager/0.log" Jul 06 19:59:08.294630 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:08.294611 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lf8hn_33a0983c-34cc-4244-99d0-7e3e69f94bbc/manager/1.log" Jul 06 19:59:08.515641 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:08.515613 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56f978bf57-w7bsg_4ca071df-c56c-4c75-bd57-f30e0b31976f/manager/0.log" Jul 06 19:59:08.556681 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:08.556586 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56f978bf57-w7bsg_4ca071df-c56c-4c75-bd57-f30e0b31976f/copy-manifests/0.log" Jul 06 19:59:08.587275 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:08.587251 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-5j9s4_f53cbc16-d0a4-4020-810f-2a09d6a2755b/postgres/0.log" Jul 06 19:59:09.425635 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:09.425606 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4rzbhr_a5e16a95-6efc-4765-9e90-1a5f17af02e3/istio-proxy/0.log" Jul 06 19:59:09.580412 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:09.580380 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fffdf4997-n76hp_b354c43c-54c3-4679-9bc3-dea447406712/kube-auth-proxy/0.log" Jul 06 19:59:09.647899 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:09.647871 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-f9c2h_91828ea8-792a-4816-a022-1b6bb2e8c19c/istio-proxy/0.log" Jul 06 19:59:09.673258 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:09.673213 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-f9c2h_91828ea8-792a-4816-a022-1b6bb2e8c19c/istio-proxy/1.log" Jul 06 19:59:09.719332 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:09.719253 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-z4dzj_11b1cacc-17e2-43f2-874b-febb142224bb/payload-pre-processing/0.log" Jul 06 19:59:09.773181 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:09.773149 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-fhccx_54d38cb0-b022-4644-a5d4-7e8e8d908019/payload-processing/0.log" Jul 06 19:59:10.096170 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:10.096141 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-tgs8p_eab4be20-1503-4405-8541-0ecb2172a74e/openshift-lws-operator/0.log" Jul 06 19:59:10.979599 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:10.979573 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4rzbhr_a5e16a95-6efc-4765-9e90-1a5f17af02e3/istio-proxy/0.log" Jul 06 19:59:11.134321 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:11.134232 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fffdf4997-n76hp_b354c43c-54c3-4679-9bc3-dea447406712/kube-auth-proxy/0.log" Jul 06 19:59:11.175651 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:11.175622 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-f9c2h_91828ea8-792a-4816-a022-1b6bb2e8c19c/istio-proxy/0.log" Jul 06 19:59:11.202645 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:11.202612 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-f9c2h_91828ea8-792a-4816-a022-1b6bb2e8c19c/istio-proxy/1.log" Jul 06 19:59:11.230208 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:11.230181 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-z4dzj_11b1cacc-17e2-43f2-874b-febb142224bb/payload-pre-processing/0.log" Jul 06 19:59:11.267160 ip-10-0-133-201 kubenswrapper[2566]: I0706 19:59:11.267128 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-fhccx_54d38cb0-b022-4644-a5d4-7e8e8d908019/payload-processing/0.log"