Jul 12 19:19:49.631296 ip-10-0-130-167 systemd[1]: Starting Kubernetes Kubelet... Jul 12 19:19:50.053203 ip-10-0-130-167 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 19:19:50.053203 ip-10-0-130-167 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 12 19:19:50.053203 ip-10-0-130-167 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 19:19:50.053203 ip-10-0-130-167 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 12 19:19:50.053203 ip-10-0-130-167 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 19:19:50.054625 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.054541 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 12 19:19:50.059939 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.059920 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:19:50.059939 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.059940 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:19:50.060035 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.059945 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:19:50.060035 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.059948 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:19:50.060035 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.059951 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:19:50.060035 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.059954 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:19:50.060035 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.059956 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:19:50.060035 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.059959 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:19:50.060035 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.059962 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:19:50.060035 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.059964 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:19:50.060035 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.059968 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:19:50.060035 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.059970 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:19:50.060035 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.059973 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:19:50.060035 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.059975 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:19:50.060035 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060000 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:19:50.060035 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060003 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:19:50.060035 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060006 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:19:50.060035 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060008 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:19:50.060035 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060011 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:19:50.060035 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060013 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:19:50.060035 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060016 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:19:50.060481 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060019 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:19:50.060481 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060023 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:19:50.060481 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060027 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:19:50.060481 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060029 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:19:50.060481 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060032 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:19:50.060481 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060035 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:19:50.060481 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060038 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:19:50.060481 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060041 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:19:50.060481 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060043 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:19:50.060481 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060046 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:19:50.060481 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060049 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:19:50.060481 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060052 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:19:50.060481 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060054 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:19:50.060481 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060057 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:19:50.060481 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060059 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:19:50.060481 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060061 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:19:50.060481 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060064 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:19:50.060481 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060066 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:19:50.060481 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060068 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:19:50.060923 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060071 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:19:50.060923 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060073 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:19:50.060923 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060076 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:19:50.060923 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060078 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:19:50.060923 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060081 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:19:50.060923 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060083 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:19:50.060923 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060085 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:19:50.060923 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060089 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:19:50.060923 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060092 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:19:50.060923 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060094 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:19:50.060923 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060097 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:19:50.060923 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060099 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:19:50.060923 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060103 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:19:50.060923 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060105 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:19:50.060923 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060109 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:19:50.060923 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060112 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:19:50.060923 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060116 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:19:50.060923 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060120 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:19:50.060923 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060123 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:19:50.060923 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060126 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:19:50.061430 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060129 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:19:50.061430 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060132 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:19:50.061430 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060134 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:19:50.061430 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060137 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:19:50.061430 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060139 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:19:50.061430 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060142 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:19:50.061430 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060145 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:19:50.061430 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060149 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:19:50.061430 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060152 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:19:50.061430 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060155 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:19:50.061430 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060158 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:19:50.061430 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060160 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:19:50.061430 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060163 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:19:50.061430 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060165 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:19:50.061430 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060168 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:19:50.061430 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060172 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:19:50.061430 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060175 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:19:50.061430 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060177 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:19:50.061430 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060180 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:19:50.061430 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060183 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:19:50.061906 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060186 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:19:50.061906 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060188 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:19:50.061906 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060191 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:19:50.061906 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060193 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:19:50.061906 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060196 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:19:50.061906 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060198 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:19:50.061906 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060201 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:19:50.061906 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060204 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:19:50.061906 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060206 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:19:50.061906 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060209 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:19:50.061906 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060211 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:19:50.061906 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060214 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:19:50.061906 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060216 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:19:50.061906 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060219 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:19:50.061906 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060221 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:19:50.061906 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060224 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:19:50.061906 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060226 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:19:50.061906 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060229 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:19:50.061906 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060231 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:19:50.061906 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060942 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:19:50.062408 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060950 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:19:50.062408 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060955 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:19:50.062408 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060958 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:19:50.062408 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060961 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:19:50.062408 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060964 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:19:50.062408 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060967 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:19:50.062408 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060971 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:19:50.062408 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060973 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:19:50.062408 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060992 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:19:50.062408 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060995 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:19:50.062408 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.060997 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:19:50.062408 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061000 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:19:50.062408 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061003 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:19:50.062408 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061006 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:19:50.062408 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061009 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:19:50.062408 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061012 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:19:50.062408 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061014 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:19:50.062408 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061017 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:19:50.062408 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061019 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:19:50.062408 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061022 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:19:50.063116 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061025 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:19:50.063116 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061027 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:19:50.063116 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061029 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:19:50.063116 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061032 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:19:50.063116 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061035 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:19:50.063116 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061037 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:19:50.063116 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061039 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:19:50.063116 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061042 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:19:50.063116 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061044 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:19:50.063116 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061047 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:19:50.063116 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061049 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:19:50.063116 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061051 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:19:50.063116 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061054 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:19:50.063116 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061056 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:19:50.063116 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061059 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:19:50.063116 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061061 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:19:50.063116 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061064 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:19:50.063116 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061066 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:19:50.063116 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061069 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:19:50.063116 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061072 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:19:50.063835 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061074 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:19:50.063835 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061079 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:19:50.063835 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061083 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:19:50.063835 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061086 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:19:50.063835 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061088 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:19:50.063835 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061091 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:19:50.063835 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061095 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:19:50.063835 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061097 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:19:50.063835 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061099 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:19:50.063835 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061102 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:19:50.063835 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061105 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:19:50.063835 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061107 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:19:50.063835 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061110 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:19:50.063835 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061112 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:19:50.063835 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061115 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:19:50.063835 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061117 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:19:50.063835 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061120 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:19:50.063835 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061122 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:19:50.063835 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061125 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:19:50.063835 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061127 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:19:50.064343 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061129 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:19:50.064343 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061132 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:19:50.064343 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061136 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:19:50.064343 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061140 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:19:50.064343 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061143 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:19:50.064343 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061145 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:19:50.064343 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061148 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:19:50.064343 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061150 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:19:50.064343 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061152 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:19:50.064343 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061155 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:19:50.064343 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061158 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:19:50.064343 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061160 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:19:50.064343 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061163 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:19:50.064343 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061166 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:19:50.064343 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061170 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:19:50.064343 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061173 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:19:50.064343 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061175 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:19:50.064343 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061177 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:19:50.064343 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061180 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:19:50.064796 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061182 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:19:50.064796 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061185 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:19:50.064796 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061190 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:19:50.064796 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061193 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:19:50.064796 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061195 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:19:50.064796 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061198 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:19:50.064796 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061200 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:19:50.064796 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061203 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:19:50.064796 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061205 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:19:50.064796 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061208 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:19:50.064796 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061211 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:19:50.064796 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061213 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:19:50.064796 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061215 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:19:50.064796 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061218 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:19:50.064796 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061220 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:19:50.064796 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061223 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:19:50.064796 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061225 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:19:50.064796 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061228 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:19:50.064796 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061230 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:19:50.064796 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061636 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:19:50.065293 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061641 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:19:50.065293 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061645 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:19:50.065293 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061648 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:19:50.065293 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061651 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:19:50.065293 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061654 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:19:50.065293 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061657 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:19:50.065293 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061660 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:19:50.065293 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061663 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:19:50.065293 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061665 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:19:50.065293 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061668 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:19:50.065293 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061670 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:19:50.065293 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061673 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:19:50.065293 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061676 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:19:50.065293 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061678 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:19:50.065293 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061683 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:19:50.065293 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061685 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:19:50.065293 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061688 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:19:50.065293 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061690 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:19:50.065293 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061693 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:19:50.065293 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061695 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:19:50.065799 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061698 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:19:50.065799 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061700 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:19:50.065799 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061702 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:19:50.065799 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061705 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:19:50.065799 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061707 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:19:50.065799 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061709 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:19:50.065799 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061712 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:19:50.065799 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061714 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:19:50.065799 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061717 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:19:50.065799 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061719 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:19:50.065799 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061722 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:19:50.065799 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061725 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:19:50.065799 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061727 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:19:50.065799 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061729 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:19:50.065799 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061732 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:19:50.065799 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061735 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:19:50.065799 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061737 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:19:50.065799 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061740 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:19:50.065799 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061742 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:19:50.066275 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061745 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:19:50.066275 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061747 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:19:50.066275 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061750 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:19:50.066275 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061752 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:19:50.066275 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061754 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:19:50.066275 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061757 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:19:50.066275 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061761 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:19:50.066275 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061766 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:19:50.066275 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061769 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:19:50.066275 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061772 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:19:50.066275 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061774 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:19:50.066275 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061777 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:19:50.066275 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061779 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:19:50.066275 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061784 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:19:50.066275 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061787 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:19:50.066275 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061790 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:19:50.066275 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061793 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:19:50.066275 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061795 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:19:50.066275 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061798 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:19:50.066731 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061800 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:19:50.066731 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061803 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:19:50.066731 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061805 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:19:50.066731 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061807 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:19:50.066731 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061810 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:19:50.066731 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061812 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:19:50.066731 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061815 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:19:50.066731 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061817 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:19:50.066731 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061820 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:19:50.066731 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061823 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:19:50.066731 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061825 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:19:50.066731 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061828 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:19:50.066731 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061830 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:19:50.066731 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061833 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:19:50.066731 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061835 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:19:50.066731 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061838 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:19:50.066731 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061840 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:19:50.066731 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061843 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:19:50.066731 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061845 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:19:50.066731 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061847 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:19:50.067354 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061853 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:19:50.067354 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061857 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:19:50.067354 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061859 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:19:50.067354 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061862 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:19:50.067354 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061865 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:19:50.067354 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061869 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:19:50.067354 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061872 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:19:50.067354 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061875 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:19:50.067354 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061877 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:19:50.067354 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061880 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:19:50.067354 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061882 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:19:50.067354 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061885 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:19:50.067354 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061887 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:19:50.067354 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061889 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:19:50.067354 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061892 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:19:50.067354 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061894 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:19:50.067354 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061896 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:19:50.067354 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061899 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:19:50.067354 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061901 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:19:50.067354 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.061904 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.061975 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.061997 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062005 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062012 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062017 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062022 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062028 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062032 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062035 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062038 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062041 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062044 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062048 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062053 2564 flags.go:64] FLAG: --cgroup-root="" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062056 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062059 2564 flags.go:64] FLAG: --client-ca-file="" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062062 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062064 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062068 2564 flags.go:64] FLAG: --cluster-domain="" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062070 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062073 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062076 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062079 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 12 19:19:50.067838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062083 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062085 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062088 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062092 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062094 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062098 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062100 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062103 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062106 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062111 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062114 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062117 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062120 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062123 2564 flags.go:64] FLAG: --enable-server="true" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062126 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062130 2564 flags.go:64] FLAG: --event-burst="100" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062132 2564 flags.go:64] FLAG: --event-qps="50" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062135 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062138 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062141 2564 flags.go:64] FLAG: --eviction-hard="" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062144 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062147 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062150 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062154 2564 flags.go:64] FLAG: --eviction-soft="" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062156 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 12 19:19:50.068425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062159 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062162 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062165 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062168 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062171 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062173 2564 flags.go:64] FLAG: --feature-gates="" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062177 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062180 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062183 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062186 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062189 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062191 2564 flags.go:64] FLAG: --help="false" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062194 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-130-167.ec2.internal" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062197 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062200 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062203 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062208 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062212 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062214 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062217 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062220 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062222 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062225 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062228 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 12 19:19:50.069082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062231 2564 flags.go:64] FLAG: --kube-reserved="" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062234 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062237 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062240 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062243 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062246 2564 flags.go:64] FLAG: --lock-file="" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062249 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062252 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062255 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062261 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062264 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062266 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062269 2564 flags.go:64] FLAG: --logging-format="text" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062272 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062275 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062278 2564 flags.go:64] FLAG: --manifest-url="" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062280 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062284 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062287 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062291 2564 flags.go:64] FLAG: --max-pods="110" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062294 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062297 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062300 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062303 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062307 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 12 19:19:50.069664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062310 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062312 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062319 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062322 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062325 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062328 2564 flags.go:64] FLAG: --pod-cidr="" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062331 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062337 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062339 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062342 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062345 2564 flags.go:64] FLAG: --port="10250" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062348 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062351 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ced3bf197b7b0a7e" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062354 2564 flags.go:64] FLAG: --qos-reserved="" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062357 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062360 2564 flags.go:64] FLAG: --register-node="true" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062363 2564 flags.go:64] FLAG: --register-with-taints="" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062367 2564 flags.go:64] FLAG: --registry-burst="10" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062369 2564 flags.go:64] FLAG: --registry-qps="5" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062372 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062375 2564 flags.go:64] FLAG: --reserved-memory="" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062378 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062381 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062384 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062387 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 12 19:19:50.070275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062389 2564 flags.go:64] FLAG: --runonce="false" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062392 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062395 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062398 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062400 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062403 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062406 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062410 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062413 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062417 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062419 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062422 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062425 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062428 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062431 2564 flags.go:64] FLAG: --system-cgroups="" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062433 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062439 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062441 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062445 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062449 2564 flags.go:64] FLAG: --tls-min-version="" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062451 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062454 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062457 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062460 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062463 2564 flags.go:64] FLAG: --v="2" Jul 12 19:19:50.070885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062467 2564 flags.go:64] FLAG: --version="false" Jul 12 19:19:50.071517 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062470 2564 flags.go:64] FLAG: --vmodule="" Jul 12 19:19:50.071517 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062474 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 12 19:19:50.071517 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.062477 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 12 19:19:50.071517 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063111 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:19:50.071517 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063131 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:19:50.071517 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063142 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:19:50.071517 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063148 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:19:50.071517 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063154 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:19:50.071517 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063159 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:19:50.071517 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063164 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:19:50.071517 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063168 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:19:50.071517 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063173 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:19:50.071517 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063177 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:19:50.071517 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063182 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:19:50.071517 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063187 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:19:50.071517 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063193 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:19:50.071517 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063198 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:19:50.071517 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063207 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:19:50.071517 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063212 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:19:50.071517 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063216 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:19:50.072030 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063221 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:19:50.072030 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063225 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:19:50.072030 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063230 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:19:50.072030 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063235 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:19:50.072030 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063240 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:19:50.072030 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063244 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:19:50.072030 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063248 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:19:50.072030 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063253 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:19:50.072030 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063257 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:19:50.072030 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063261 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:19:50.072030 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063270 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:19:50.072030 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063274 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:19:50.072030 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063279 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:19:50.072030 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063284 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:19:50.072030 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063288 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:19:50.072030 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063293 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:19:50.072030 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063299 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:19:50.072030 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063306 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:19:50.072030 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063312 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:19:50.072516 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063318 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:19:50.072516 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063323 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:19:50.072516 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063328 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:19:50.072516 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063338 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:19:50.072516 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063342 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:19:50.072516 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063347 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:19:50.072516 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063351 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:19:50.072516 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063355 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:19:50.072516 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063359 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:19:50.072516 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063364 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:19:50.072516 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063369 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:19:50.072516 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063373 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:19:50.072516 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063378 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:19:50.072516 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063382 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:19:50.072516 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063386 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:19:50.072516 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063391 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:19:50.072516 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063401 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:19:50.072516 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063405 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:19:50.072516 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063409 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:19:50.072516 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063414 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:19:50.073000 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063418 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:19:50.073000 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063423 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:19:50.073000 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063427 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:19:50.073000 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063431 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:19:50.073000 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063435 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:19:50.073000 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063440 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:19:50.073000 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063444 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:19:50.073000 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063449 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:19:50.073000 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063453 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:19:50.073000 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063462 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:19:50.073000 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063468 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:19:50.073000 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063474 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:19:50.073000 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063479 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:19:50.073000 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063483 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:19:50.073000 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063488 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:19:50.073000 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063492 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:19:50.073000 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063497 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:19:50.073000 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063502 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:19:50.073000 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063506 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:19:50.073000 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063513 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:19:50.073479 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063523 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:19:50.073479 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063527 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:19:50.073479 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063532 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:19:50.073479 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063588 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:19:50.073479 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063596 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:19:50.073479 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063603 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:19:50.073479 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063608 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:19:50.073479 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063613 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:19:50.073479 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063618 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:19:50.073479 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063623 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:19:50.073479 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063628 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:19:50.073479 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063632 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:19:50.073479 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063637 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:19:50.073479 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063642 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:19:50.073479 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063647 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:19:50.073479 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063673 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:19:50.073479 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063704 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:19:50.073479 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063831 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:19:50.073479 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063840 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:19:50.073940 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063844 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:19:50.073940 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063851 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:19:50.073940 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063856 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:19:50.073940 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.063859 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:19:50.073940 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.064599 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 19:19:50.073940 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.071514 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 12 19:19:50.073940 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.071532 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 12 19:19:50.073940 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.071552 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 12 19:19:50.073940 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.071558 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 12 19:19:50.073940 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071616 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:19:50.073940 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071621 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:19:50.073940 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071624 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:19:50.073940 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071627 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:19:50.073940 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071630 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:19:50.074295 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071633 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:19:50.074295 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071636 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:19:50.074295 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071638 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:19:50.074295 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071641 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:19:50.074295 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071643 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:19:50.074295 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071646 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:19:50.074295 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071648 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:19:50.074295 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071651 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:19:50.074295 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071653 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:19:50.074295 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071656 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:19:50.074295 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071658 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:19:50.074295 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071661 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:19:50.074295 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071663 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:19:50.074295 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071666 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:19:50.074295 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071669 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:19:50.074295 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071671 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:19:50.074295 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071674 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:19:50.074295 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071677 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:19:50.074295 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071679 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:19:50.074295 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071681 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:19:50.074771 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071684 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:19:50.074771 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071688 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:19:50.074771 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071691 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:19:50.074771 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071694 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:19:50.074771 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071698 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:19:50.074771 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071701 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:19:50.074771 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071704 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:19:50.074771 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071707 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:19:50.074771 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071710 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:19:50.074771 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071713 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:19:50.074771 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071715 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:19:50.074771 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071718 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:19:50.074771 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071720 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:19:50.074771 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071723 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:19:50.074771 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071725 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:19:50.074771 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071728 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:19:50.074771 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071730 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:19:50.074771 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071733 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:19:50.074771 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071735 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:19:50.074771 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071738 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:19:50.075265 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071740 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:19:50.075265 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071743 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:19:50.075265 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071745 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:19:50.075265 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071748 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:19:50.075265 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071750 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:19:50.075265 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071752 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:19:50.075265 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071756 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:19:50.075265 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071759 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:19:50.075265 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071763 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:19:50.075265 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071765 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:19:50.075265 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071767 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:19:50.075265 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071770 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:19:50.075265 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071772 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:19:50.075265 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071775 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:19:50.075265 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071777 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:19:50.075265 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071780 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:19:50.075265 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071783 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:19:50.075265 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071786 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:19:50.075265 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071788 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:19:50.075759 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071791 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:19:50.075759 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071794 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:19:50.075759 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071797 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:19:50.075759 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071799 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:19:50.075759 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071802 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:19:50.075759 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071804 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:19:50.075759 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071807 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:19:50.075759 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071844 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:19:50.075759 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071848 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:19:50.075759 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071851 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:19:50.075759 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071854 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:19:50.075759 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071856 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:19:50.075759 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071859 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:19:50.075759 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071861 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:19:50.075759 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071864 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:19:50.075759 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071866 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:19:50.075759 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071868 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:19:50.075759 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071871 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:19:50.075759 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071873 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:19:50.076251 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071876 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:19:50.076251 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071878 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:19:50.076251 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071881 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:19:50.076251 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071884 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:19:50.076251 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071887 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:19:50.076251 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071889 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:19:50.076251 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071891 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:19:50.076251 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071896 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:19:50.076251 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071899 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:19:50.076251 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071902 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:19:50.076251 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071905 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:19:50.076251 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071908 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:19:50.076251 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071910 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:19:50.076251 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071913 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:19:50.076251 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071915 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:19:50.076251 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.071918 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:19:50.076635 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.071923 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 19:19:50.076635 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072056 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:19:50.076635 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072060 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:19:50.076635 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072063 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:19:50.076635 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072067 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:19:50.076635 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072070 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:19:50.076635 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072073 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:19:50.076635 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072075 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:19:50.076635 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072078 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:19:50.076635 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072080 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:19:50.076635 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072083 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:19:50.076635 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072085 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:19:50.076635 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072087 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:19:50.076635 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072090 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:19:50.076635 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072092 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:19:50.077013 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072095 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:19:50.077013 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072097 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:19:50.077013 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072100 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:19:50.077013 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072103 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:19:50.077013 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072105 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:19:50.077013 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072108 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:19:50.077013 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072110 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:19:50.077013 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072113 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:19:50.077013 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072115 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:19:50.077013 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072117 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:19:50.077013 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072120 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:19:50.077013 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072123 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:19:50.077013 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072126 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:19:50.077013 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072128 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:19:50.077013 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072131 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:19:50.077013 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072134 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:19:50.077013 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072136 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:19:50.077013 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072139 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:19:50.077013 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072141 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:19:50.077013 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072144 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:19:50.077498 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072146 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:19:50.077498 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072149 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:19:50.077498 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072151 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:19:50.077498 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072154 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:19:50.077498 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072156 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:19:50.077498 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072158 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:19:50.077498 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072162 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:19:50.077498 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072166 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:19:50.077498 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072168 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:19:50.077498 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072171 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:19:50.077498 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072174 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:19:50.077498 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072177 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:19:50.077498 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072180 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:19:50.077498 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072182 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:19:50.077498 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072185 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:19:50.077498 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072187 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:19:50.077498 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072190 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:19:50.077498 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072192 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:19:50.077498 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072195 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:19:50.077961 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072198 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:19:50.077961 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072200 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:19:50.077961 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072203 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:19:50.077961 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072206 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:19:50.077961 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072210 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:19:50.077961 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072213 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:19:50.077961 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072216 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:19:50.077961 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072219 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:19:50.077961 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072222 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:19:50.077961 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072224 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:19:50.077961 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072227 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:19:50.077961 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072229 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:19:50.077961 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072231 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:19:50.077961 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072234 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:19:50.077961 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072236 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:19:50.077961 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072239 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:19:50.077961 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072241 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:19:50.077961 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072244 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:19:50.077961 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072246 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:19:50.078422 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072249 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:19:50.078422 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072251 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:19:50.078422 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072254 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:19:50.078422 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072256 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:19:50.078422 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072259 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:19:50.078422 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072261 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:19:50.078422 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072264 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:19:50.078422 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072266 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:19:50.078422 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072268 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:19:50.078422 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072271 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:19:50.078422 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072274 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:19:50.078422 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072278 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:19:50.078422 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072281 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:19:50.078422 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072284 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:19:50.078422 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072286 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:19:50.078422 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072289 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:19:50.078422 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072291 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:19:50.078422 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072294 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:19:50.078422 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072296 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:19:50.078881 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072299 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:19:50.078881 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072301 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:19:50.078881 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072304 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:19:50.078881 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072306 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:19:50.078881 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072309 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:19:50.078881 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072311 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:19:50.078881 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072313 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:19:50.078881 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:50.072316 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:19:50.078881 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.072321 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 19:19:50.078881 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.072430 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 12 19:19:50.078881 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.075240 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 12 19:19:50.078881 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.076689 2564 server.go:1019] "Starting client certificate rotation" Jul 12 19:19:50.078881 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.076786 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 12 19:19:50.078881 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.076822 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 12 19:19:50.080260 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.080245 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 12 19:19:50.083950 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.083931 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 12 19:19:50.095385 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.095363 2564 log.go:25] "Validated CRI v1 runtime API" Jul 12 19:19:50.098377 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.098364 2564 log.go:25] "Validated CRI v1 image API" Jul 12 19:19:50.099264 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.099251 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 12 19:19:50.106035 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.106012 2564 fs.go:135] Filesystem UUIDs: map[1d07a77a-eba8-400c-a364-30a849be4187:/dev/nvme0n1p4 296ce107-bf00-443a-b996-ff8babb1f811:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 12 19:19:50.106132 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.106032 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 12 19:19:50.112091 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.111952 2564 manager.go:216] Machine: {Timestamp:2026-07-12 19:19:50.111000651 +0000 UTC m=+0.359313724 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098070 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25fb762dd04b8f7ebf5391a3656831 SystemUUID:ec25fb76-2dd0-4b8f-7ebf-5391a3656831 BootID:340eec0c-c3df-4307-82a2-5f2a8a352ee7 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3c:16:63:d9:59 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3c:16:63:d9:59 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:ce:e9:1f:28:b8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 12 19:19:50.112091 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.112076 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 12 19:19:50.112230 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.112161 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 12 19:19:50.113722 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.113704 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 12 19:19:50.115384 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.115344 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 12 19:19:50.115532 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.115393 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-167.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 12 19:19:50.116172 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.116161 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 12 19:19:50.116211 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.116174 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 12 19:19:50.116211 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.116182 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 12 19:19:50.116211 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.116193 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 12 19:19:50.117101 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.117091 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 12 19:19:50.120507 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.120496 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 12 19:19:50.120640 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.120631 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 12 19:19:50.120672 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.120662 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 12 19:19:50.120672 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.120671 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 12 19:19:50.120722 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.120688 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 12 19:19:50.120722 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.120696 2564 kubelet.go:398] "Adding apiserver pod source" Jul 12 19:19:50.120722 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.120705 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 12 19:19:50.121124 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.121112 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 12 19:19:50.121174 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.121133 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 12 19:19:50.124038 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.124022 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 12 19:19:50.125438 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.125424 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 12 19:19:50.125539 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.125462 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 12 19:19:50.126159 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.126145 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 12 19:19:50.126225 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.126165 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 12 19:19:50.126225 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.126174 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 12 19:19:50.126225 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.126182 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 12 19:19:50.126225 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.126193 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 12 19:19:50.126225 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.126202 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 12 19:19:50.126411 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.126284 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 12 19:19:50.126411 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.126300 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 12 19:19:50.126411 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.126311 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 12 19:19:50.126411 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.126320 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 12 19:19:50.126411 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.126332 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 12 19:19:50.126411 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.126344 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 12 19:19:50.127213 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.127202 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 12 19:19:50.127271 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.127216 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 12 19:19:50.130625 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.130608 2564 server.go:1268] "Started kubelet" Jul 12 19:19:50.130711 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.130664 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 12 19:19:50.130871 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.130761 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 12 19:19:50.130952 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.130874 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-167.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 12 19:19:50.130952 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.130909 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 12 19:19:50.130952 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.130950 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 12 19:19:50.131153 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.131136 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-167.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 12 19:19:50.131725 ip-10-0-130-167 systemd[1]: Started Kubernetes Kubelet. Jul 12 19:19:50.132431 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.132369 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 12 19:19:50.132877 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.132864 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 12 19:19:50.135403 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.135385 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 12 19:19:50.138402 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.138383 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 12 19:19:50.138490 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.138404 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 12 19:19:50.138543 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.138509 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-167.ec2.internal\" not found" Jul 12 19:19:50.138946 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.137910 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-167.ec2.internal.18c1a109a03d5019 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-167.ec2.internal,UID:ip-10-0-130-167.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-167.ec2.internal,},FirstTimestamp:2026-07-12 19:19:50.130577433 +0000 UTC m=+0.378890506,LastTimestamp:2026-07-12 19:19:50.130577433 +0000 UTC m=+0.378890506,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-167.ec2.internal,}" Jul 12 19:19:50.139159 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.139136 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 12 19:19:50.139269 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.139253 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 12 19:19:50.139348 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.139158 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 12 19:19:50.139411 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.139392 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 12 19:19:50.139411 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.139401 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 12 19:19:50.139638 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.139619 2564 factory.go:55] Registering systemd factory Jul 12 19:19:50.139689 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.139641 2564 factory.go:223] Registration of the systemd container factory successfully Jul 12 19:19:50.139900 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.139885 2564 factory.go:153] Registering CRI-O factory Jul 12 19:19:50.139900 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.139900 2564 factory.go:223] Registration of the crio container factory successfully Jul 12 19:19:50.140035 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.139959 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 12 19:19:50.140035 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.140000 2564 factory.go:103] Registering Raw factory Jul 12 19:19:50.140035 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.140014 2564 manager.go:1196] Started watching for new ooms in manager Jul 12 19:19:50.140742 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.140729 2564 manager.go:349] Starting recovery of all containers Jul 12 19:19:50.146282 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.144452 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-167.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 12 19:19:50.146282 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.144518 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 12 19:19:50.152217 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.152051 2564 manager.go:354] Recovery completed Jul 12 19:19:50.156831 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.156813 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:19:50.158712 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.158690 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:19:50.158801 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.158730 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:19:50.158801 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.158741 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:19:50.159389 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.159371 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 12 19:19:50.159389 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.159387 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 12 19:19:50.159509 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.159404 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 12 19:19:50.159509 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.159388 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:19:50.160185 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.160171 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:19:50.160243 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.160200 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:19:50.160243 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.160213 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:19:50.160342 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.160326 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v2hq2" Jul 12 19:19:50.160821 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.160807 2564 policy_none.go:49] "None policy: Start" Jul 12 19:19:50.160874 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.160828 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 12 19:19:50.160874 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.160843 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 12 19:19:50.161263 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.161186 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-167.ec2.internal.18c1a109a1eaa26e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-167.ec2.internal,UID:ip-10-0-130-167.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-167.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-167.ec2.internal,},FirstTimestamp:2026-07-12 19:19:50.158713454 +0000 UTC m=+0.407026534,LastTimestamp:2026-07-12 19:19:50.158713454 +0000 UTC m=+0.407026534,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-167.ec2.internal,}" Jul 12 19:19:50.162181 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.162167 2564 policy_none.go:47] "Start" Jul 12 19:19:50.167894 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.167874 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v2hq2" Jul 12 19:19:50.169849 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.169779 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-167.ec2.internal.18c1a109a1eaf974 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-167.ec2.internal,UID:ip-10-0-130-167.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-167.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-167.ec2.internal,},FirstTimestamp:2026-07-12 19:19:50.158735732 +0000 UTC m=+0.407048804,LastTimestamp:2026-07-12 19:19:50.158735732 +0000 UTC m=+0.407048804,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-167.ec2.internal,}" Jul 12 19:19:50.171920 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.171860 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-167.ec2.internal.18c1a109a1eb1ed4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-167.ec2.internal,UID:ip-10-0-130-167.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-167.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-167.ec2.internal,},FirstTimestamp:2026-07-12 19:19:50.1587453 +0000 UTC m=+0.407058372,LastTimestamp:2026-07-12 19:19:50.1587453 +0000 UTC m=+0.407058372,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-167.ec2.internal,}" Jul 12 19:19:50.201560 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.201543 2564 manager.go:337] "Starting Device Plugin manager" Jul 12 19:19:50.201642 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.201577 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 12 19:19:50.201642 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.201587 2564 server.go:87] "Starting device plugin registration server" Jul 12 19:19:50.201821 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.201808 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 12 19:19:50.201865 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.201841 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 12 19:19:50.201942 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.201926 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 12 19:19:50.202043 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.202031 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 12 19:19:50.202086 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.202044 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 12 19:19:50.202603 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.202582 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 12 19:19:50.202678 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.202627 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-167.ec2.internal\" not found" Jul 12 19:19:50.253423 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.253387 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 12 19:19:50.254885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.254863 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 12 19:19:50.254968 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.254904 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 12 19:19:50.254968 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.254960 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 12 19:19:50.255096 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.255019 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 12 19:19:50.257331 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.257312 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:19:50.302220 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.302193 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:19:50.303332 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.303281 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:19:50.303332 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.303316 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:19:50.303332 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.303329 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:19:50.303490 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.303371 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-167.ec2.internal" Jul 12 19:19:50.312232 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.312216 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-167.ec2.internal" Jul 12 19:19:50.312331 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.312238 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-167.ec2.internal\": node \"ip-10-0-130-167.ec2.internal\" not found" Jul 12 19:19:50.332317 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.332284 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-167.ec2.internal\" not found" Jul 12 19:19:50.355357 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.355328 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-167.ec2.internal"] Jul 12 19:19:50.355465 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.355406 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:19:50.356452 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.356437 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:19:50.356537 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.356471 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:19:50.356537 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.356486 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:19:50.357941 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.357925 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:19:50.358102 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.358088 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal" Jul 12 19:19:50.358148 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.358116 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:19:50.358793 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.358779 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:19:50.358857 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.358803 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:19:50.358857 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.358813 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:19:50.359083 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.359069 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:19:50.359155 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.359106 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:19:50.359155 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.359120 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:19:50.360248 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.360234 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-167.ec2.internal" Jul 12 19:19:50.360311 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.360261 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:19:50.360942 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.360922 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:19:50.361034 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.360953 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:19:50.361034 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.360964 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:19:50.393593 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.393565 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-167.ec2.internal\" not found" node="ip-10-0-130-167.ec2.internal" Jul 12 19:19:50.397851 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.397834 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-167.ec2.internal\" not found" node="ip-10-0-130-167.ec2.internal" Jul 12 19:19:50.432435 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.432404 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-167.ec2.internal\" not found" Jul 12 19:19:50.441239 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.441220 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72c5a76d2f244265b5a44f7b0562c46f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal\" (UID: \"72c5a76d2f244265b5a44f7b0562c46f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal" Jul 12 19:19:50.441296 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.441248 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72c5a76d2f244265b5a44f7b0562c46f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal\" (UID: \"72c5a76d2f244265b5a44f7b0562c46f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal" Jul 12 19:19:50.441296 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.441273 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c9f6b081b0c599a6d0bb3dfcb98d7c97-config\") pod \"kube-apiserver-proxy-ip-10-0-130-167.ec2.internal\" (UID: \"c9f6b081b0c599a6d0bb3dfcb98d7c97\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-167.ec2.internal" Jul 12 19:19:50.533431 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.533384 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-167.ec2.internal\" not found" Jul 12 19:19:50.541721 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.541693 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c9f6b081b0c599a6d0bb3dfcb98d7c97-config\") pod \"kube-apiserver-proxy-ip-10-0-130-167.ec2.internal\" (UID: \"c9f6b081b0c599a6d0bb3dfcb98d7c97\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-167.ec2.internal" Jul 12 19:19:50.541769 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.541726 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72c5a76d2f244265b5a44f7b0562c46f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal\" (UID: \"72c5a76d2f244265b5a44f7b0562c46f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal" Jul 12 19:19:50.541769 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.541742 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72c5a76d2f244265b5a44f7b0562c46f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal\" (UID: \"72c5a76d2f244265b5a44f7b0562c46f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal" Jul 12 19:19:50.541842 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.541784 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72c5a76d2f244265b5a44f7b0562c46f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal\" (UID: \"72c5a76d2f244265b5a44f7b0562c46f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal" Jul 12 19:19:50.541842 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.541798 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c9f6b081b0c599a6d0bb3dfcb98d7c97-config\") pod \"kube-apiserver-proxy-ip-10-0-130-167.ec2.internal\" (UID: \"c9f6b081b0c599a6d0bb3dfcb98d7c97\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-167.ec2.internal" Jul 12 19:19:50.541842 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.541816 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72c5a76d2f244265b5a44f7b0562c46f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal\" (UID: \"72c5a76d2f244265b5a44f7b0562c46f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal" Jul 12 19:19:50.634255 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.634186 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-167.ec2.internal\" not found" Jul 12 19:19:50.696749 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.696711 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal" Jul 12 19:19:50.701116 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:50.700911 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-167.ec2.internal" Jul 12 19:19:50.735255 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.735220 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-167.ec2.internal\" not found" Jul 12 19:19:50.835775 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:50.835727 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-167.ec2.internal\" not found" Jul 12 19:19:52.591293 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:52.591272 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-167.ec2.internal\" not found" Jul 12 19:19:52.692036 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:52.691987 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-167.ec2.internal\" not found" Jul 12 19:19:52.731506 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:52.731484 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 12 19:19:52.732009 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:52.731636 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 12 19:19:52.792283 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:52.792258 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-167.ec2.internal\" not found" Jul 12 19:19:52.827807 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:52.827771 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-11 19:14:51 +0000 UTC" deadline="2028-03-21 06:13:54.595522032 +0000 UTC" Jul 12 19:19:52.827807 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:52.827799 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14818h54m1.767726302s" Jul 12 19:19:52.854977 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:52.854918 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:19:52.856505 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:52.856485 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:19:52.856670 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:52.856518 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:19:52.856670 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:52.856532 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:19:52.893089 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:52.893062 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-167.ec2.internal\" not found" Jul 12 19:19:52.924145 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:52.924120 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:19:52.950776 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:52.950744 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:19:52.993953 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:52.993920 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal" Jul 12 19:19:53.006531 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.006507 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 12 19:19:53.008412 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.008396 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-167.ec2.internal" Jul 12 19:19:53.016526 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.016505 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 12 19:19:53.048604 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:53.048572 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f6b081b0c599a6d0bb3dfcb98d7c97.slice/crio-aa3c186799f21e306971acef99dfc2b5ff8a13ae0447099cce18128b00beff3d WatchSource:0}: Error finding container aa3c186799f21e306971acef99dfc2b5ff8a13ae0447099cce18128b00beff3d: Status 404 returned error can't find the container with id aa3c186799f21e306971acef99dfc2b5ff8a13ae0447099cce18128b00beff3d Jul 12 19:19:53.049400 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:53.049381 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c5a76d2f244265b5a44f7b0562c46f.slice/crio-ae2dba9b2e313b646e4dfd6ed863e214acb820f02fe5a8c3575a65583c33eb34 WatchSource:0}: Error finding container ae2dba9b2e313b646e4dfd6ed863e214acb820f02fe5a8c3575a65583c33eb34: Status 404 returned error can't find the container with id ae2dba9b2e313b646e4dfd6ed863e214acb820f02fe5a8c3575a65583c33eb34 Jul 12 19:19:53.055630 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.055605 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 19:19:53.303745 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.303716 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:19:53.676728 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.676701 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:19:53.777393 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.777348 2564 apiserver.go:52] "Watching apiserver" Jul 12 19:19:53.786741 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.786709 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 12 19:19:53.787281 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.787257 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-grbbl","openshift-image-registry/node-ca-thbcj","openshift-multus/multus-additional-cni-plugins-xv5j2","openshift-multus/multus-fdm9d","openshift-network-operator/iptables-alerter-l5rxz","openshift-ovn-kubernetes/ovnkube-node-jj2hs","kube-system/konnectivity-agent-g4vrt","kube-system/kube-apiserver-proxy-ip-10-0-130-167.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal","openshift-multus/network-metrics-daemon-fz99s","openshift-network-diagnostics/network-check-target-5gknd","kube-system/global-pull-secret-syncer-nd2b6","openshift-cluster-node-tuning-operator/tuned-bsltc"] Jul 12 19:19:53.789050 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.789011 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.791135 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.791109 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l5rxz" Jul 12 19:19:53.791688 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.791636 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 12 19:19:53.791859 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.791814 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 12 19:19:53.791927 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.791880 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 12 19:19:53.791962 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.791817 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 12 19:19:53.792138 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.792115 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g8nj9\"" Jul 12 19:19:53.792257 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.792242 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 12 19:19:53.793107 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.793088 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 12 19:19:53.793436 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.793404 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bvf8d\"" Jul 12 19:19:53.797099 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.797079 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 12 19:19:53.797366 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.797348 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g4vrt" Jul 12 19:19:53.798424 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.798392 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.799471 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.799452 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 12 19:19:53.799864 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.799843 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 12 19:19:53.799947 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.799898 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 12 19:19:53.800006 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.799995 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hlcb7\"" Jul 12 19:19:53.800135 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.800114 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 12 19:19:53.800391 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.800374 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-thbcj" Jul 12 19:19:53.801258 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.801233 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 12 19:19:53.801258 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.801244 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kjbw5\"" Jul 12 19:19:53.801610 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.801587 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.802527 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.802484 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 12 19:19:53.803367 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.803348 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gww8\"" Jul 12 19:19:53.803669 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.803648 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 12 19:19:53.803775 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.803688 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 12 19:19:53.804866 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.804010 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 12 19:19:53.804866 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.804351 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-grbbl" Jul 12 19:19:53.804866 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.804543 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xf5gf\"" Jul 12 19:19:53.804866 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.804855 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 12 19:19:53.805557 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.804894 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 12 19:19:53.805557 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.805066 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 12 19:19:53.805557 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.805101 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 12 19:19:53.805557 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.805152 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 12 19:19:53.805889 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.805863 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:53.806341 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.806317 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-cnibin\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.806425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.806354 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-os-release\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.806425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.806380 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.806425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.806402 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4kwv\" (UniqueName: \"kubernetes.io/projected/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-kube-api-access-n4kwv\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.806560 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.806428 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a3e4a12-84ce-4021-8966-a0b8d3517fdc-iptables-alerter-script\") pod \"iptables-alerter-l5rxz\" (UID: \"1a3e4a12-84ce-4021-8966-a0b8d3517fdc\") " pod="openshift-network-operator/iptables-alerter-l5rxz" Jul 12 19:19:53.806560 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.806453 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-host-var-lib-cni-multus\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.806560 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.806473 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-etc-kubernetes\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.806560 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.806497 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ab20a71c-5305-4984-8c11-e3b5e5451fcb-serviceca\") pod \"node-ca-thbcj\" (UID: \"ab20a71c-5305-4984-8c11-e3b5e5451fcb\") " pod="openshift-image-registry/node-ca-thbcj" Jul 12 19:19:53.806560 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.806518 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-kubelet\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.806560 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.806540 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-var-lib-openvswitch\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.806828 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.806597 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-run-openvswitch\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.806828 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.806620 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-cnibin\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.806828 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.806642 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-host-run-netns\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.806828 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.806666 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czmx7\" (UniqueName: \"kubernetes.io/projected/ea178531-de7c-44c8-911b-308836d78022-kube-api-access-czmx7\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.806828 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.806691 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5rrc\" (UniqueName: \"kubernetes.io/projected/ab20a71c-5305-4984-8c11-e3b5e5451fcb-kube-api-access-w5rrc\") pod \"node-ca-thbcj\" (UID: \"ab20a71c-5305-4984-8c11-e3b5e5451fcb\") " pod="openshift-image-registry/node-ca-thbcj" Jul 12 19:19:53.806828 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.806757 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 12 19:19:53.807004 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.806757 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-run-systemd\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.807004 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.806963 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.807404 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807009 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c1cc216-5111-43d9-979f-ad6cc1beece6-ovnkube-config\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.807404 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807206 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c1cc216-5111-43d9-979f-ad6cc1beece6-env-overrides\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.807404 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807231 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.807404 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807247 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ea178531-de7c-44c8-911b-308836d78022-multus-daemon-config\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.807404 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807287 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-node-log\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.807404 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807311 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-cni-netd\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.807404 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807339 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a7a4e516-ce2a-4b83-8d99-83c61331242f-konnectivity-ca\") pod \"konnectivity-agent-g4vrt\" (UID: \"a7a4e516-ce2a-4b83-8d99-83c61331242f\") " pod="kube-system/konnectivity-agent-g4vrt" Jul 12 19:19:53.807404 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807355 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-os-release\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.807404 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807379 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmmqw\" (UniqueName: \"kubernetes.io/projected/1a3e4a12-84ce-4021-8966-a0b8d3517fdc-kube-api-access-tmmqw\") pod \"iptables-alerter-l5rxz\" (UID: \"1a3e4a12-84ce-4021-8966-a0b8d3517fdc\") " pod="openshift-network-operator/iptables-alerter-l5rxz" Jul 12 19:19:53.807956 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807408 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a3e4a12-84ce-4021-8966-a0b8d3517fdc-host-slash\") pod \"iptables-alerter-l5rxz\" (UID: \"1a3e4a12-84ce-4021-8966-a0b8d3517fdc\") " pod="openshift-network-operator/iptables-alerter-l5rxz" Jul 12 19:19:53.807956 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807443 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab20a71c-5305-4984-8c11-e3b5e5451fcb-host\") pod \"node-ca-thbcj\" (UID: \"ab20a71c-5305-4984-8c11-e3b5e5451fcb\") " pod="openshift-image-registry/node-ca-thbcj" Jul 12 19:19:53.807956 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807471 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-run-netns\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.807956 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807498 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-run-ovn-kubernetes\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.807956 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807523 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-cni-bin\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.807956 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807548 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-system-cni-dir\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.807956 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807584 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-hostroot\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.807956 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807614 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-multus-conf-dir\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.807956 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807643 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-systemd-units\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.807956 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807678 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-slash\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.807956 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-log-socket\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.807956 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807765 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpxqs\" (UniqueName: \"kubernetes.io/projected/0c1cc216-5111-43d9-979f-ad6cc1beece6-kube-api-access-jpxqs\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.807956 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807803 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.807956 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807828 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-system-cni-dir\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.807956 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807865 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea178531-de7c-44c8-911b-308836d78022-cni-binary-copy\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.807956 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807896 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-host-run-k8s-cni-cncf-io\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.807956 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807923 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-host-var-lib-kubelet\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.808759 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807949 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-host-run-multus-certs\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.808759 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807961 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:19:53.808759 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.807975 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.808759 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.808001 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a7a4e516-ce2a-4b83-8d99-83c61331242f-agent-certs\") pod \"konnectivity-agent-g4vrt\" (UID: \"a7a4e516-ce2a-4b83-8d99-83c61331242f\") " pod="kube-system/konnectivity-agent-g4vrt" Jul 12 19:19:53.808759 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.808043 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-host-var-lib-cni-bin\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.808759 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:53.808048 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz99s" podUID="8a0fd497-acc1-4391-bb37-3a00c2c3362d" Jul 12 19:19:53.808759 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.808066 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-etc-openvswitch\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.808759 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.808090 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-run-ovn\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.808759 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.808112 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c1cc216-5111-43d9-979f-ad6cc1beece6-ovn-node-metrics-cert\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.808759 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.808135 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c1cc216-5111-43d9-979f-ad6cc1beece6-ovnkube-script-lib\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.808759 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.808170 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-multus-cni-dir\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.808759 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.808205 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-multus-socket-dir-parent\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.809254 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.809230 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hz8mp\"" Jul 12 19:19:53.809359 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.809345 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 12 19:19:53.809548 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.809530 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 12 19:19:53.809618 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.809580 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 12 19:19:53.809618 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.809587 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:19:53.809719 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.809645 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5b8j7\"" Jul 12 19:19:53.809719 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:53.809650 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gknd" podUID="a2c5b5b8-f936-4c30-99f8-9cc04474d080" Jul 12 19:19:53.809719 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.809589 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 12 19:19:53.810961 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.810943 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:19:53.811071 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:53.811002 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd2b6" podUID="697ae350-ebe5-4859-a7d4-389fd1da28f8" Jul 12 19:19:53.811611 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.811444 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 12 19:19:53.812356 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.812337 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:53.814533 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.814514 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 12 19:19:53.814533 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.814525 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g95ld\"" Jul 12 19:19:53.814683 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.814666 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 12 19:19:53.828578 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.828524 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-npl66" Jul 12 19:19:53.838483 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.838462 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-npl66" Jul 12 19:19:53.896302 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.896271 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 12 19:19:53.908458 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908426 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-run-openvswitch\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.908667 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908468 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-node-log\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.908667 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908499 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c0ce3d01-8fce-4587-9c82-f8673d917c05-device-dir\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:53.908667 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908525 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c1cc216-5111-43d9-979f-ad6cc1beece6-ovn-node-metrics-cert\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.908667 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908551 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a3e4a12-84ce-4021-8966-a0b8d3517fdc-iptables-alerter-script\") pod \"iptables-alerter-l5rxz\" (UID: \"1a3e4a12-84ce-4021-8966-a0b8d3517fdc\") " pod="openshift-network-operator/iptables-alerter-l5rxz" Jul 12 19:19:53.908667 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908574 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-var-lib-openvswitch\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.908667 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908599 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.908667 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908627 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25c9d\" (UniqueName: \"kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d\") pod \"network-check-target-5gknd\" (UID: \"a2c5b5b8-f936-4c30-99f8-9cc04474d080\") " pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:19:53.908667 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908651 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-run-systemd\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.909070 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908675 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a01d1fa-13e1-47af-b3af-2a9de7431dd1-tmp-dir\") pod \"node-resolver-grbbl\" (UID: \"9a01d1fa-13e1-47af-b3af-2a9de7431dd1\") " pod="openshift-dns/node-resolver-grbbl" Jul 12 19:19:53.909070 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908700 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.909070 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908772 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-sys\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:53.909070 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908796 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-929fj\" (UniqueName: \"kubernetes.io/projected/9a01d1fa-13e1-47af-b3af-2a9de7431dd1-kube-api-access-929fj\") pod \"node-resolver-grbbl\" (UID: \"9a01d1fa-13e1-47af-b3af-2a9de7431dd1\") " pod="openshift-dns/node-resolver-grbbl" Jul 12 19:19:53.909070 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908819 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmmqw\" (UniqueName: \"kubernetes.io/projected/1a3e4a12-84ce-4021-8966-a0b8d3517fdc-kube-api-access-tmmqw\") pod \"iptables-alerter-l5rxz\" (UID: \"1a3e4a12-84ce-4021-8966-a0b8d3517fdc\") " pod="openshift-network-operator/iptables-alerter-l5rxz" Jul 12 19:19:53.909070 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908853 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a3e4a12-84ce-4021-8966-a0b8d3517fdc-host-slash\") pod \"iptables-alerter-l5rxz\" (UID: \"1a3e4a12-84ce-4021-8966-a0b8d3517fdc\") " pod="openshift-network-operator/iptables-alerter-l5rxz" Jul 12 19:19:53.909070 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908878 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab20a71c-5305-4984-8c11-e3b5e5451fcb-host\") pod \"node-ca-thbcj\" (UID: \"ab20a71c-5305-4984-8c11-e3b5e5451fcb\") " pod="openshift-image-registry/node-ca-thbcj" Jul 12 19:19:53.909070 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908901 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-run-netns\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.909070 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908925 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-sysctl-conf\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:53.909070 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908947 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-var-lib-kubelet\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:53.909070 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908969 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c0ce3d01-8fce-4587-9c82-f8673d917c05-socket-dir\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:53.909070 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909006 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-system-cni-dir\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.909070 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909047 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-hostroot\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.909665 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909076 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-node-log\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.909665 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909078 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-log-socket\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.909665 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909122 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-log-socket\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.909665 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909140 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpxqs\" (UniqueName: \"kubernetes.io/projected/0c1cc216-5111-43d9-979f-ad6cc1beece6-kube-api-access-jpxqs\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.909665 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909181 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret\") pod \"global-pull-secret-syncer-nd2b6\" (UID: \"697ae350-ebe5-4859-a7d4-389fd1da28f8\") " pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:19:53.909665 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea178531-de7c-44c8-911b-308836d78022-cni-binary-copy\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.909665 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909241 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-host-var-lib-kubelet\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.909665 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909267 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-host-run-multus-certs\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.909665 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909291 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9a01d1fa-13e1-47af-b3af-2a9de7431dd1-hosts-file\") pod \"node-resolver-grbbl\" (UID: \"9a01d1fa-13e1-47af-b3af-2a9de7431dd1\") " pod="openshift-dns/node-resolver-grbbl" Jul 12 19:19:53.909665 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909314 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-sysctl-d\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:53.909665 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909337 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-run\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:53.909665 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909359 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4b25a35-4484-420e-8a26-dbfafbe0ac30-tmp\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:53.909665 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909384 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c0ce3d01-8fce-4587-9c82-f8673d917c05-registration-dir\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:53.909665 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909396 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a3e4a12-84ce-4021-8966-a0b8d3517fdc-host-slash\") pod \"iptables-alerter-l5rxz\" (UID: \"1a3e4a12-84ce-4021-8966-a0b8d3517fdc\") " pod="openshift-network-operator/iptables-alerter-l5rxz" Jul 12 19:19:53.909665 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909406 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c0ce3d01-8fce-4587-9c82-f8673d917c05-sys-fs\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:53.909665 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909441 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-multus-socket-dir-parent\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.909665 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909469 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-cnibin\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.910477 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909493 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-os-release\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.910477 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909521 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh9cz\" (UniqueName: \"kubernetes.io/projected/8a0fd497-acc1-4391-bb37-3a00c2c3362d-kube-api-access-gh9cz\") pod \"network-metrics-daemon-fz99s\" (UID: \"8a0fd497-acc1-4391-bb37-3a00c2c3362d\") " pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:19:53.910477 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909550 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/697ae350-ebe5-4859-a7d4-389fd1da28f8-kubelet-config\") pod \"global-pull-secret-syncer-nd2b6\" (UID: \"697ae350-ebe5-4859-a7d4-389fd1da28f8\") " pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:19:53.910477 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909579 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c1cc216-5111-43d9-979f-ad6cc1beece6-ovnkube-script-lib\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.910477 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909606 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-host-var-lib-cni-multus\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.910477 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909631 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-etc-kubernetes\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.910477 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909650 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-var-lib-openvswitch\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.910477 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909659 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a3e4a12-84ce-4021-8966-a0b8d3517fdc-iptables-alerter-script\") pod \"iptables-alerter-l5rxz\" (UID: \"1a3e4a12-84ce-4021-8966-a0b8d3517fdc\") " pod="openshift-network-operator/iptables-alerter-l5rxz" Jul 12 19:19:53.910477 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909700 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-kubelet\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.910477 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909742 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.910477 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909751 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-run-netns\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.910477 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909786 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab20a71c-5305-4984-8c11-e3b5e5451fcb-host\") pod \"node-ca-thbcj\" (UID: \"ab20a71c-5305-4984-8c11-e3b5e5451fcb\") " pod="openshift-image-registry/node-ca-thbcj" Jul 12 19:19:53.910477 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909823 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-system-cni-dir\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.910477 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.908987 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 12 19:19:53.910477 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909840 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-run-systemd\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.910477 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909924 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-multus-socket-dir-parent\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.910477 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909960 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-cnibin\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.911290 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910008 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-os-release\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.911290 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910084 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-etc-kubernetes\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.911290 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910140 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-host-var-lib-cni-multus\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.911290 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910169 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-host-var-lib-kubelet\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.911290 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910182 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea178531-de7c-44c8-911b-308836d78022-cni-binary-copy\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.911290 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910192 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-host-run-multus-certs\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.911290 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.909655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-kubelet\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.911290 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910245 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-run-openvswitch\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.911290 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910286 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-tuned\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:53.911290 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910311 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/697ae350-ebe5-4859-a7d4-389fd1da28f8-dbus\") pod \"global-pull-secret-syncer-nd2b6\" (UID: \"697ae350-ebe5-4859-a7d4-389fd1da28f8\") " pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:19:53.911290 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910345 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-cnibin\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.911290 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910370 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-host-run-netns\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.911290 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910394 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czmx7\" (UniqueName: \"kubernetes.io/projected/ea178531-de7c-44c8-911b-308836d78022-kube-api-access-czmx7\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.911290 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910418 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5rrc\" (UniqueName: \"kubernetes.io/projected/ab20a71c-5305-4984-8c11-e3b5e5451fcb-kube-api-access-w5rrc\") pod \"node-ca-thbcj\" (UID: \"ab20a71c-5305-4984-8c11-e3b5e5451fcb\") " pod="openshift-image-registry/node-ca-thbcj" Jul 12 19:19:53.911290 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910466 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c1cc216-5111-43d9-979f-ad6cc1beece6-ovnkube-config\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.911290 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910489 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c1cc216-5111-43d9-979f-ad6cc1beece6-env-overrides\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.911290 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910511 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-modprobe-d\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:53.911290 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910534 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-kubernetes\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:53.912125 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910556 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ea178531-de7c-44c8-911b-308836d78022-multus-daemon-config\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.912125 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910580 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-cni-netd\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.912125 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910591 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c1cc216-5111-43d9-979f-ad6cc1beece6-ovnkube-script-lib\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.912125 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910608 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-systemd\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:53.912125 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910681 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-host\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:53.912125 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910706 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c0ce3d01-8fce-4587-9c82-f8673d917c05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:53.912125 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910730 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs\") pod \"network-metrics-daemon-fz99s\" (UID: \"8a0fd497-acc1-4391-bb37-3a00c2c3362d\") " pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:19:53.912125 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910757 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a7a4e516-ce2a-4b83-8d99-83c61331242f-konnectivity-ca\") pod \"konnectivity-agent-g4vrt\" (UID: \"a7a4e516-ce2a-4b83-8d99-83c61331242f\") " pod="kube-system/konnectivity-agent-g4vrt" Jul 12 19:19:53.912125 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910781 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-os-release\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.912125 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910805 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-run-ovn-kubernetes\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.912125 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910855 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-cni-bin\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.912125 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910882 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-sysconfig\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:53.912125 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910906 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c0ce3d01-8fce-4587-9c82-f8673d917c05-etc-selinux\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:53.912125 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910939 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-multus-conf-dir\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.912125 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910965 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-systemd-units\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.912125 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.910992 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-slash\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.912125 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911107 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chdzw\" (UniqueName: \"kubernetes.io/projected/c0ce3d01-8fce-4587-9c82-f8673d917c05-kube-api-access-chdzw\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:53.912885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911115 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-run-ovn-kubernetes\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.912885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911137 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.912885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911166 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-system-cni-dir\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.912885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911172 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-cni-bin\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.912885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911181 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c1cc216-5111-43d9-979f-ad6cc1beece6-env-overrides\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.912885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911191 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-host-run-k8s-cni-cncf-io\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.912885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911216 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-lib-modules\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:53.912885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911230 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-cnibin\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.912885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911260 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.912885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911268 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-cni-netd\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.912885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911290 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a7a4e516-ce2a-4b83-8d99-83c61331242f-agent-certs\") pod \"konnectivity-agent-g4vrt\" (UID: \"a7a4e516-ce2a-4b83-8d99-83c61331242f\") " pod="kube-system/konnectivity-agent-g4vrt" Jul 12 19:19:53.912885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911317 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-host-var-lib-cni-bin\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.912885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911363 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-etc-openvswitch\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.912885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911392 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-run-ovn\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.912885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911407 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-hostroot\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.912885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911417 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-466kr\" (UniqueName: \"kubernetes.io/projected/a4b25a35-4484-420e-8a26-dbfafbe0ac30-kube-api-access-466kr\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:53.912885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911450 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-multus-cni-dir\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.913543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911479 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.913543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911528 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4kwv\" (UniqueName: \"kubernetes.io/projected/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-kube-api-access-n4kwv\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.913543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911532 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ea178531-de7c-44c8-911b-308836d78022-multus-daemon-config\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.913543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911554 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ab20a71c-5305-4984-8c11-e3b5e5451fcb-serviceca\") pod \"node-ca-thbcj\" (UID: \"ab20a71c-5305-4984-8c11-e3b5e5451fcb\") " pod="openshift-image-registry/node-ca-thbcj" Jul 12 19:19:53.913543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911575 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-host-slash\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.913543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.911498 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-host-run-netns\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.913543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.912050 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c1cc216-5111-43d9-979f-ad6cc1beece6-ovnkube-config\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.913543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.912076 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ab20a71c-5305-4984-8c11-e3b5e5451fcb-serviceca\") pod \"node-ca-thbcj\" (UID: \"ab20a71c-5305-4984-8c11-e3b5e5451fcb\") " pod="openshift-image-registry/node-ca-thbcj" Jul 12 19:19:53.913543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.912108 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-system-cni-dir\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.913543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.912124 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-multus-cni-dir\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.913543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.912134 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-host-run-k8s-cni-cncf-io\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.913543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.912134 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-etc-openvswitch\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.913543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.912159 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-run-ovn\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.913543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.912163 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-host-var-lib-cni-bin\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.913543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.912207 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-multus-conf-dir\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.913543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.912308 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.913543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.912374 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ea178531-de7c-44c8-911b-308836d78022-os-release\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.913543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.912451 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c1cc216-5111-43d9-979f-ad6cc1beece6-systemd-units\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.914169 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.912465 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.914169 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.912548 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.914169 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.912657 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.914169 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.912867 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a7a4e516-ce2a-4b83-8d99-83c61331242f-konnectivity-ca\") pod \"konnectivity-agent-g4vrt\" (UID: \"a7a4e516-ce2a-4b83-8d99-83c61331242f\") " pod="kube-system/konnectivity-agent-g4vrt" Jul 12 19:19:53.914412 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.914392 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c1cc216-5111-43d9-979f-ad6cc1beece6-ovn-node-metrics-cert\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:53.917416 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.916965 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a7a4e516-ce2a-4b83-8d99-83c61331242f-agent-certs\") pod \"konnectivity-agent-g4vrt\" (UID: \"a7a4e516-ce2a-4b83-8d99-83c61331242f\") " pod="kube-system/konnectivity-agent-g4vrt" Jul 12 19:19:53.918816 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.918796 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmmqw\" (UniqueName: \"kubernetes.io/projected/1a3e4a12-84ce-4021-8966-a0b8d3517fdc-kube-api-access-tmmqw\") pod \"iptables-alerter-l5rxz\" (UID: \"1a3e4a12-84ce-4021-8966-a0b8d3517fdc\") " pod="openshift-network-operator/iptables-alerter-l5rxz" Jul 12 19:19:53.919014 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.918898 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czmx7\" (UniqueName: \"kubernetes.io/projected/ea178531-de7c-44c8-911b-308836d78022-kube-api-access-czmx7\") pod \"multus-fdm9d\" (UID: \"ea178531-de7c-44c8-911b-308836d78022\") " pod="openshift-multus/multus-fdm9d" Jul 12 19:19:53.919093 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.919009 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-167.ec2.internal" event={"ID":"c9f6b081b0c599a6d0bb3dfcb98d7c97","Type":"ContainerStarted","Data":"aa3c186799f21e306971acef99dfc2b5ff8a13ae0447099cce18128b00beff3d"} Jul 12 19:19:53.921665 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.921571 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal" event={"ID":"72c5a76d2f244265b5a44f7b0562c46f","Type":"ContainerStarted","Data":"ae2dba9b2e313b646e4dfd6ed863e214acb820f02fe5a8c3575a65583c33eb34"} Jul 12 19:19:53.923971 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.923949 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5rrc\" (UniqueName: \"kubernetes.io/projected/ab20a71c-5305-4984-8c11-e3b5e5451fcb-kube-api-access-w5rrc\") pod \"node-ca-thbcj\" (UID: \"ab20a71c-5305-4984-8c11-e3b5e5451fcb\") " pod="openshift-image-registry/node-ca-thbcj" Jul 12 19:19:53.924074 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.923973 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4kwv\" (UniqueName: \"kubernetes.io/projected/39bf2054-7b11-434a-a6e3-4d1f04cdd1a3-kube-api-access-n4kwv\") pod \"multus-additional-cni-plugins-xv5j2\" (UID: \"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3\") " pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:53.928681 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:53.928569 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpxqs\" (UniqueName: \"kubernetes.io/projected/0c1cc216-5111-43d9-979f-ad6cc1beece6-kube-api-access-jpxqs\") pod \"ovnkube-node-jj2hs\" (UID: \"0c1cc216-5111-43d9-979f-ad6cc1beece6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:54.015627 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.012511 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-kubernetes\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.015627 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.012568 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-systemd\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.015627 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.012598 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-host\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.015627 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.012597 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-kubernetes\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.015627 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.012650 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-systemd\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.015627 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.012662 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-host\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.015627 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.012701 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c0ce3d01-8fce-4587-9c82-f8673d917c05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:54.015627 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.012739 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs\") pod \"network-metrics-daemon-fz99s\" (UID: \"8a0fd497-acc1-4391-bb37-3a00c2c3362d\") " pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:19:54.015627 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.012771 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-sysconfig\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.015627 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.012794 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c0ce3d01-8fce-4587-9c82-f8673d917c05-etc-selinux\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:54.015627 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.012829 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chdzw\" (UniqueName: \"kubernetes.io/projected/c0ce3d01-8fce-4587-9c82-f8673d917c05-kube-api-access-chdzw\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:54.015627 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.012858 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-lib-modules\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.015627 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.012927 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-466kr\" (UniqueName: \"kubernetes.io/projected/a4b25a35-4484-420e-8a26-dbfafbe0ac30-kube-api-access-466kr\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.015627 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.012999 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c0ce3d01-8fce-4587-9c82-f8673d917c05-device-dir\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:54.015627 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.013054 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25c9d\" (UniqueName: \"kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d\") pod \"network-check-target-5gknd\" (UID: \"a2c5b5b8-f936-4c30-99f8-9cc04474d080\") " pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:19:54.015627 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.013088 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a01d1fa-13e1-47af-b3af-2a9de7431dd1-tmp-dir\") pod \"node-resolver-grbbl\" (UID: \"9a01d1fa-13e1-47af-b3af-2a9de7431dd1\") " pod="openshift-dns/node-resolver-grbbl" Jul 12 19:19:54.015627 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.013119 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-sys\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.016508 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.013183 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-929fj\" (UniqueName: \"kubernetes.io/projected/9a01d1fa-13e1-47af-b3af-2a9de7431dd1-kube-api-access-929fj\") pod \"node-resolver-grbbl\" (UID: \"9a01d1fa-13e1-47af-b3af-2a9de7431dd1\") " pod="openshift-dns/node-resolver-grbbl" Jul 12 19:19:54.016508 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.013216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-sysctl-conf\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.016508 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.013244 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-var-lib-kubelet\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.016508 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.013304 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c0ce3d01-8fce-4587-9c82-f8673d917c05-socket-dir\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:54.016508 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.013359 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret\") pod \"global-pull-secret-syncer-nd2b6\" (UID: \"697ae350-ebe5-4859-a7d4-389fd1da28f8\") " pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:19:54.016508 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.013387 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9a01d1fa-13e1-47af-b3af-2a9de7431dd1-hosts-file\") pod \"node-resolver-grbbl\" (UID: \"9a01d1fa-13e1-47af-b3af-2a9de7431dd1\") " pod="openshift-dns/node-resolver-grbbl" Jul 12 19:19:54.016508 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.013416 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-sysctl-d\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.016508 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.013468 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-run\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.016508 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.013613 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4b25a35-4484-420e-8a26-dbfafbe0ac30-tmp\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.016508 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.013642 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c0ce3d01-8fce-4587-9c82-f8673d917c05-registration-dir\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:54.016508 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.013669 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c0ce3d01-8fce-4587-9c82-f8673d917c05-sys-fs\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:54.016508 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.013727 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9cz\" (UniqueName: \"kubernetes.io/projected/8a0fd497-acc1-4391-bb37-3a00c2c3362d-kube-api-access-gh9cz\") pod \"network-metrics-daemon-fz99s\" (UID: \"8a0fd497-acc1-4391-bb37-3a00c2c3362d\") " pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:19:54.016508 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.013760 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/697ae350-ebe5-4859-a7d4-389fd1da28f8-kubelet-config\") pod \"global-pull-secret-syncer-nd2b6\" (UID: \"697ae350-ebe5-4859-a7d4-389fd1da28f8\") " pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:19:54.016508 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.013795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-tuned\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.016508 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.013826 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/697ae350-ebe5-4859-a7d4-389fd1da28f8-dbus\") pod \"global-pull-secret-syncer-nd2b6\" (UID: \"697ae350-ebe5-4859-a7d4-389fd1da28f8\") " pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:19:54.016508 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.013876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-modprobe-d\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.016508 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.014163 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-modprobe-d\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.017202 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.014224 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-var-lib-kubelet\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.017202 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.014275 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-sysctl-conf\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.017202 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.014348 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c0ce3d01-8fce-4587-9c82-f8673d917c05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:54.017202 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.014384 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c0ce3d01-8fce-4587-9c82-f8673d917c05-socket-dir\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:54.017202 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:54.014470 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:19:54.017202 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.014482 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c0ce3d01-8fce-4587-9c82-f8673d917c05-etc-selinux\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:54.017202 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:54.014720 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs podName:8a0fd497-acc1-4391-bb37-3a00c2c3362d nodeName:}" failed. No retries permitted until 2026-07-12 19:19:54.514691881 +0000 UTC m=+3.107965974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs") pod "network-metrics-daemon-fz99s" (UID: "8a0fd497-acc1-4391-bb37-3a00c2c3362d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:19:54.017202 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.014907 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-lib-modules\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.017202 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:54.015044 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:19:54.017202 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:54.015113 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret podName:697ae350-ebe5-4859-a7d4-389fd1da28f8 nodeName:}" failed. No retries permitted until 2026-07-12 19:19:54.51508365 +0000 UTC m=+3.108357728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret") pod "global-pull-secret-syncer-nd2b6" (UID: "697ae350-ebe5-4859-a7d4-389fd1da28f8") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:19:54.017202 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.015180 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c0ce3d01-8fce-4587-9c82-f8673d917c05-sys-fs\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:54.017202 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.015237 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a01d1fa-13e1-47af-b3af-2a9de7431dd1-tmp-dir\") pod \"node-resolver-grbbl\" (UID: \"9a01d1fa-13e1-47af-b3af-2a9de7431dd1\") " pod="openshift-dns/node-resolver-grbbl" Jul 12 19:19:54.017202 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.015261 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/697ae350-ebe5-4859-a7d4-389fd1da28f8-kubelet-config\") pod \"global-pull-secret-syncer-nd2b6\" (UID: \"697ae350-ebe5-4859-a7d4-389fd1da28f8\") " pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:19:54.017202 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.015376 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-sysconfig\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.017202 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.015434 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9a01d1fa-13e1-47af-b3af-2a9de7431dd1-hosts-file\") pod \"node-resolver-grbbl\" (UID: \"9a01d1fa-13e1-47af-b3af-2a9de7431dd1\") " pod="openshift-dns/node-resolver-grbbl" Jul 12 19:19:54.017202 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.015509 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-run\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.017202 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.015572 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/697ae350-ebe5-4859-a7d4-389fd1da28f8-dbus\") pod \"global-pull-secret-syncer-nd2b6\" (UID: \"697ae350-ebe5-4859-a7d4-389fd1da28f8\") " pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:19:54.018108 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.015239 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-sys\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.018108 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.015765 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c0ce3d01-8fce-4587-9c82-f8673d917c05-registration-dir\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:54.018108 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.015830 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c0ce3d01-8fce-4587-9c82-f8673d917c05-device-dir\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:54.018108 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.015945 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-sysctl-d\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.018312 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.018155 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4b25a35-4484-420e-8a26-dbfafbe0ac30-tmp\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.018312 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.018210 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4b25a35-4484-420e-8a26-dbfafbe0ac30-etc-tuned\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.021398 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:54.021250 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:19:54.021398 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:54.021276 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:19:54.021398 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:54.021291 2564 projected.go:196] Error preparing data for projected volume kube-api-access-25c9d for pod openshift-network-diagnostics/network-check-target-5gknd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:19:54.021398 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:54.021364 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d podName:a2c5b5b8-f936-4c30-99f8-9cc04474d080 nodeName:}" failed. No retries permitted until 2026-07-12 19:19:54.521346961 +0000 UTC m=+3.114621052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-25c9d" (UniqueName: "kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d") pod "network-check-target-5gknd" (UID: "a2c5b5b8-f936-4c30-99f8-9cc04474d080") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:19:54.023697 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.023671 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chdzw\" (UniqueName: \"kubernetes.io/projected/c0ce3d01-8fce-4587-9c82-f8673d917c05-kube-api-access-chdzw\") pod \"aws-ebs-csi-driver-node-tr475\" (UID: \"c0ce3d01-8fce-4587-9c82-f8673d917c05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:54.023946 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.023921 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-929fj\" (UniqueName: \"kubernetes.io/projected/9a01d1fa-13e1-47af-b3af-2a9de7431dd1-kube-api-access-929fj\") pod \"node-resolver-grbbl\" (UID: \"9a01d1fa-13e1-47af-b3af-2a9de7431dd1\") " pod="openshift-dns/node-resolver-grbbl" Jul 12 19:19:54.025237 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.025212 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-466kr\" (UniqueName: \"kubernetes.io/projected/a4b25a35-4484-420e-8a26-dbfafbe0ac30-kube-api-access-466kr\") pod \"tuned-bsltc\" (UID: \"a4b25a35-4484-420e-8a26-dbfafbe0ac30\") " pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.025833 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.025814 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh9cz\" (UniqueName: \"kubernetes.io/projected/8a0fd497-acc1-4391-bb37-3a00c2c3362d-kube-api-access-gh9cz\") pod \"network-metrics-daemon-fz99s\" (UID: \"8a0fd497-acc1-4391-bb37-3a00c2c3362d\") " pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:19:54.108761 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.108723 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xv5j2" Jul 12 19:19:54.118331 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.118305 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l5rxz" Jul 12 19:19:54.118699 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:54.118671 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39bf2054_7b11_434a_a6e3_4d1f04cdd1a3.slice/crio-3c27eb1b190cc94e84a6fb2903b8280f81578a3ea843aee62196c4bff2bdcfc6 WatchSource:0}: Error finding container 3c27eb1b190cc94e84a6fb2903b8280f81578a3ea843aee62196c4bff2bdcfc6: Status 404 returned error can't find the container with id 3c27eb1b190cc94e84a6fb2903b8280f81578a3ea843aee62196c4bff2bdcfc6 Jul 12 19:19:54.125601 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:54.125575 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3e4a12_84ce_4021_8966_a0b8d3517fdc.slice/crio-86c39e873894cfbd5e2bb439aa1a6af3128bccf0a6cc532418f2397914eab59d WatchSource:0}: Error finding container 86c39e873894cfbd5e2bb439aa1a6af3128bccf0a6cc532418f2397914eab59d: Status 404 returned error can't find the container with id 86c39e873894cfbd5e2bb439aa1a6af3128bccf0a6cc532418f2397914eab59d Jul 12 19:19:54.128689 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.128670 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g4vrt" Jul 12 19:19:54.132810 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.132785 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fdm9d" Jul 12 19:19:54.140754 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.140734 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-thbcj" Jul 12 19:19:54.146540 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.146517 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:19:54.153483 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.153461 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-grbbl" Jul 12 19:19:54.163395 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.163378 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" Jul 12 19:19:54.172437 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.172412 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bsltc" Jul 12 19:19:54.517575 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.517539 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret\") pod \"global-pull-secret-syncer-nd2b6\" (UID: \"697ae350-ebe5-4859-a7d4-389fd1da28f8\") " pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:19:54.517738 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.517604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs\") pod \"network-metrics-daemon-fz99s\" (UID: \"8a0fd497-acc1-4391-bb37-3a00c2c3362d\") " pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:19:54.517738 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:54.517680 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:19:54.517738 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:54.517701 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:19:54.517868 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:54.517763 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret podName:697ae350-ebe5-4859-a7d4-389fd1da28f8 nodeName:}" failed. No retries permitted until 2026-07-12 19:19:55.517744417 +0000 UTC m=+4.111018512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret") pod "global-pull-secret-syncer-nd2b6" (UID: "697ae350-ebe5-4859-a7d4-389fd1da28f8") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:19:54.517868 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:54.517780 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs podName:8a0fd497-acc1-4391-bb37-3a00c2c3362d nodeName:}" failed. No retries permitted until 2026-07-12 19:19:55.517772513 +0000 UTC m=+4.111046589 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs") pod "network-metrics-daemon-fz99s" (UID: "8a0fd497-acc1-4391-bb37-3a00c2c3362d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:19:54.618212 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.618175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25c9d\" (UniqueName: \"kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d\") pod \"network-check-target-5gknd\" (UID: \"a2c5b5b8-f936-4c30-99f8-9cc04474d080\") " pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:19:54.618383 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:54.618323 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:19:54.618383 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:54.618346 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:19:54.618383 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:54.618358 2564 projected.go:196] Error preparing data for projected volume kube-api-access-25c9d for pod openshift-network-diagnostics/network-check-target-5gknd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:19:54.618509 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:54.618417 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d podName:a2c5b5b8-f936-4c30-99f8-9cc04474d080 nodeName:}" failed. No retries permitted until 2026-07-12 19:19:55.618402884 +0000 UTC m=+4.211676971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-25c9d" (UniqueName: "kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d") pod "network-check-target-5gknd" (UID: "a2c5b5b8-f936-4c30-99f8-9cc04474d080") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:19:54.840823 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.840700 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-11 19:14:53 +0000 UTC" deadline="2027-12-29 05:55:24.846212526 +0000 UTC" Jul 12 19:19:54.840823 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.840778 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12826h35m30.005440082s" Jul 12 19:19:54.924857 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.924821 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xv5j2" event={"ID":"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3","Type":"ContainerStarted","Data":"3c27eb1b190cc94e84a6fb2903b8280f81578a3ea843aee62196c4bff2bdcfc6"} Jul 12 19:19:54.928338 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:54.928293 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l5rxz" event={"ID":"1a3e4a12-84ce-4021-8966-a0b8d3517fdc","Type":"ContainerStarted","Data":"86c39e873894cfbd5e2bb439aa1a6af3128bccf0a6cc532418f2397914eab59d"} Jul 12 19:19:55.241722 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:55.241501 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ce3d01_8fce_4587_9c82_f8673d917c05.slice/crio-8840b95abfcf34c71af40649190c59e56df15ca144bac55a8e2da1df47b3524e WatchSource:0}: Error finding container 8840b95abfcf34c71af40649190c59e56df15ca144bac55a8e2da1df47b3524e: Status 404 returned error can't find the container with id 8840b95abfcf34c71af40649190c59e56df15ca144bac55a8e2da1df47b3524e Jul 12 19:19:55.242979 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:55.242932 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a4e516_ce2a_4b83_8d99_83c61331242f.slice/crio-4579f8054bc46967ac69ce11821b74b0b7e526b13c821d0273a4bd9a3a91cada WatchSource:0}: Error finding container 4579f8054bc46967ac69ce11821b74b0b7e526b13c821d0273a4bd9a3a91cada: Status 404 returned error can't find the container with id 4579f8054bc46967ac69ce11821b74b0b7e526b13c821d0273a4bd9a3a91cada Jul 12 19:19:55.266158 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:19:55.266130 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a01d1fa_13e1_47af_b3af_2a9de7431dd1.slice/crio-eea4669b1a3d8e0bfffc4e70c9ea7964f634eac28926c1ede5c211b30461d45f WatchSource:0}: Error finding container eea4669b1a3d8e0bfffc4e70c9ea7964f634eac28926c1ede5c211b30461d45f: Status 404 returned error can't find the container with id eea4669b1a3d8e0bfffc4e70c9ea7964f634eac28926c1ede5c211b30461d45f Jul 12 19:19:55.526004 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:55.525879 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret\") pod \"global-pull-secret-syncer-nd2b6\" (UID: \"697ae350-ebe5-4859-a7d4-389fd1da28f8\") " pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:19:55.526004 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:55.525963 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs\") pod \"network-metrics-daemon-fz99s\" (UID: \"8a0fd497-acc1-4391-bb37-3a00c2c3362d\") " pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:19:55.526214 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:55.526082 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:19:55.526214 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:55.526112 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:19:55.526214 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:55.526143 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret podName:697ae350-ebe5-4859-a7d4-389fd1da28f8 nodeName:}" failed. No retries permitted until 2026-07-12 19:19:57.526129334 +0000 UTC m=+6.119403410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret") pod "global-pull-secret-syncer-nd2b6" (UID: "697ae350-ebe5-4859-a7d4-389fd1da28f8") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:19:55.526214 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:55.526163 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs podName:8a0fd497-acc1-4391-bb37-3a00c2c3362d nodeName:}" failed. No retries permitted until 2026-07-12 19:19:57.526151225 +0000 UTC m=+6.119425301 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs") pod "network-metrics-daemon-fz99s" (UID: "8a0fd497-acc1-4391-bb37-3a00c2c3362d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:19:55.627465 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:55.626864 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25c9d\" (UniqueName: \"kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d\") pod \"network-check-target-5gknd\" (UID: \"a2c5b5b8-f936-4c30-99f8-9cc04474d080\") " pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:19:55.627465 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:55.627055 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:19:55.627465 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:55.627072 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:19:55.627465 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:55.627081 2564 projected.go:196] Error preparing data for projected volume kube-api-access-25c9d for pod openshift-network-diagnostics/network-check-target-5gknd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:19:55.627465 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:55.627137 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d podName:a2c5b5b8-f936-4c30-99f8-9cc04474d080 nodeName:}" failed. No retries permitted until 2026-07-12 19:19:57.627122427 +0000 UTC m=+6.220396505 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-25c9d" (UniqueName: "kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d") pod "network-check-target-5gknd" (UID: "a2c5b5b8-f936-4c30-99f8-9cc04474d080") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:19:55.846211 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:55.846118 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-11 19:14:53 +0000 UTC" deadline="2027-12-28 09:04:51.13693934 +0000 UTC" Jul 12 19:19:55.846211 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:55.846162 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12805h44m55.290781996s" Jul 12 19:19:55.913727 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:55.913696 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:19:55.913900 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:55.913854 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz99s" podUID="8a0fd497-acc1-4391-bb37-3a00c2c3362d" Jul 12 19:19:55.914347 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:55.914327 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:19:55.914505 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:55.914484 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gknd" podUID="a2c5b5b8-f936-4c30-99f8-9cc04474d080" Jul 12 19:19:55.914577 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:55.914563 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:19:55.914658 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:55.914639 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd2b6" podUID="697ae350-ebe5-4859-a7d4-389fd1da28f8" Jul 12 19:19:55.944897 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:55.944858 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" event={"ID":"0c1cc216-5111-43d9-979f-ad6cc1beece6","Type":"ContainerStarted","Data":"65ddb540b3303c9e87dde99ec2ff44c38ff3808b2566818470f03086ef0da3ba"} Jul 12 19:19:55.948669 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:55.948598 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fdm9d" event={"ID":"ea178531-de7c-44c8-911b-308836d78022","Type":"ContainerStarted","Data":"1aa4349ff3ce50560a61a75b614ab982d57d05d8e6f0b213282f47bca003a688"} Jul 12 19:19:55.954014 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:55.953962 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-thbcj" event={"ID":"ab20a71c-5305-4984-8c11-e3b5e5451fcb","Type":"ContainerStarted","Data":"37ff5a40696238c3c646cddb4786aaa5c4d1a4df83d1f43af228bf2172d8ee9c"} Jul 12 19:19:55.960134 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:55.960062 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g4vrt" event={"ID":"a7a4e516-ce2a-4b83-8d99-83c61331242f","Type":"ContainerStarted","Data":"4579f8054bc46967ac69ce11821b74b0b7e526b13c821d0273a4bd9a3a91cada"} Jul 12 19:19:55.963838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:55.963810 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" event={"ID":"c0ce3d01-8fce-4587-9c82-f8673d917c05","Type":"ContainerStarted","Data":"8840b95abfcf34c71af40649190c59e56df15ca144bac55a8e2da1df47b3524e"} Jul 12 19:19:55.970901 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:55.970874 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-167.ec2.internal" event={"ID":"c9f6b081b0c599a6d0bb3dfcb98d7c97","Type":"ContainerStarted","Data":"3ec8ba8b6955514b36aa5069d8fa8cf16528f006d5fa6258d0c17658baaf5ad5"} Jul 12 19:19:55.977723 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:55.977677 2564 generic.go:356] "Generic (PLEG): container finished" podID="72c5a76d2f244265b5a44f7b0562c46f" containerID="b0c4335c1df57075d6c3f79d840690660d772773de2d0b6eeee9b1c3c987c7d2" exitCode=0 Jul 12 19:19:55.977827 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:55.977758 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal" event={"ID":"72c5a76d2f244265b5a44f7b0562c46f","Type":"ContainerDied","Data":"b0c4335c1df57075d6c3f79d840690660d772773de2d0b6eeee9b1c3c987c7d2"} Jul 12 19:19:55.984681 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:55.984626 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-167.ec2.internal" podStartSLOduration=2.984608304 podStartE2EDuration="2.984608304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:19:55.984047665 +0000 UTC m=+4.577321765" watchObservedRunningTime="2026-07-12 19:19:55.984608304 +0000 UTC m=+4.577882404" Jul 12 19:19:55.987293 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:55.987229 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bsltc" event={"ID":"a4b25a35-4484-420e-8a26-dbfafbe0ac30","Type":"ContainerStarted","Data":"959355d8d78c0e6288c60acf39c3c2bb6be81fed9dd72fab7a74403a9e34453e"} Jul 12 19:19:55.991091 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:55.991041 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-grbbl" event={"ID":"9a01d1fa-13e1-47af-b3af-2a9de7431dd1","Type":"ContainerStarted","Data":"eea4669b1a3d8e0bfffc4e70c9ea7964f634eac28926c1ede5c211b30461d45f"} Jul 12 19:19:57.544562 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:57.544528 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs\") pod \"network-metrics-daemon-fz99s\" (UID: \"8a0fd497-acc1-4391-bb37-3a00c2c3362d\") " pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:19:57.545115 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:57.544610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret\") pod \"global-pull-secret-syncer-nd2b6\" (UID: \"697ae350-ebe5-4859-a7d4-389fd1da28f8\") " pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:19:57.545115 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:57.544734 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:19:57.545115 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:57.544791 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret podName:697ae350-ebe5-4859-a7d4-389fd1da28f8 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:01.544772928 +0000 UTC m=+10.138047004 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret") pod "global-pull-secret-syncer-nd2b6" (UID: "697ae350-ebe5-4859-a7d4-389fd1da28f8") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:19:57.545115 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:57.544855 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:19:57.545115 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:57.544889 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs podName:8a0fd497-acc1-4391-bb37-3a00c2c3362d nodeName:}" failed. No retries permitted until 2026-07-12 19:20:01.544878705 +0000 UTC m=+10.138152786 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs") pod "network-metrics-daemon-fz99s" (UID: "8a0fd497-acc1-4391-bb37-3a00c2c3362d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:19:57.646182 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:57.645556 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25c9d\" (UniqueName: \"kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d\") pod \"network-check-target-5gknd\" (UID: \"a2c5b5b8-f936-4c30-99f8-9cc04474d080\") " pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:19:57.646182 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:57.645713 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:19:57.646182 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:57.645735 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:19:57.646182 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:57.645750 2564 projected.go:196] Error preparing data for projected volume kube-api-access-25c9d for pod openshift-network-diagnostics/network-check-target-5gknd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:19:57.646182 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:57.645811 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d podName:a2c5b5b8-f936-4c30-99f8-9cc04474d080 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:01.645792743 +0000 UTC m=+10.239066838 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-25c9d" (UniqueName: "kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d") pod "network-check-target-5gknd" (UID: "a2c5b5b8-f936-4c30-99f8-9cc04474d080") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:19:57.912202 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:57.912164 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:19:57.912352 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:57.912164 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:19:57.912352 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:57.912316 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz99s" podUID="8a0fd497-acc1-4391-bb37-3a00c2c3362d" Jul 12 19:19:57.912459 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:57.912384 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gknd" podUID="a2c5b5b8-f936-4c30-99f8-9cc04474d080" Jul 12 19:19:57.912497 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:57.912468 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:19:57.912565 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:57.912543 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd2b6" podUID="697ae350-ebe5-4859-a7d4-389fd1da28f8" Jul 12 19:19:59.913909 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:59.913108 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:19:59.913909 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:59.913234 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gknd" podUID="a2c5b5b8-f936-4c30-99f8-9cc04474d080" Jul 12 19:19:59.913909 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:59.913303 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:19:59.913909 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:59.913371 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd2b6" podUID="697ae350-ebe5-4859-a7d4-389fd1da28f8" Jul 12 19:19:59.913909 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:19:59.913423 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:19:59.913909 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:19:59.913498 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz99s" podUID="8a0fd497-acc1-4391-bb37-3a00c2c3362d" Jul 12 19:20:01.577839 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:01.577798 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret\") pod \"global-pull-secret-syncer-nd2b6\" (UID: \"697ae350-ebe5-4859-a7d4-389fd1da28f8\") " pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:20:01.578233 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:01.577871 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs\") pod \"network-metrics-daemon-fz99s\" (UID: \"8a0fd497-acc1-4391-bb37-3a00c2c3362d\") " pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:20:01.578233 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:01.578003 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:20:01.578233 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:01.578089 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs podName:8a0fd497-acc1-4391-bb37-3a00c2c3362d nodeName:}" failed. No retries permitted until 2026-07-12 19:20:09.578069538 +0000 UTC m=+18.171343622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs") pod "network-metrics-daemon-fz99s" (UID: "8a0fd497-acc1-4391-bb37-3a00c2c3362d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:20:01.578450 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:01.578425 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:20:01.578558 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:01.578472 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret podName:697ae350-ebe5-4859-a7d4-389fd1da28f8 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:09.578459008 +0000 UTC m=+18.171733084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret") pod "global-pull-secret-syncer-nd2b6" (UID: "697ae350-ebe5-4859-a7d4-389fd1da28f8") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:20:01.679782 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:01.679209 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25c9d\" (UniqueName: \"kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d\") pod \"network-check-target-5gknd\" (UID: \"a2c5b5b8-f936-4c30-99f8-9cc04474d080\") " pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:20:01.679782 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:01.679357 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:20:01.679782 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:01.679374 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:20:01.679782 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:01.679386 2564 projected.go:196] Error preparing data for projected volume kube-api-access-25c9d for pod openshift-network-diagnostics/network-check-target-5gknd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:20:01.679782 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:01.679445 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d podName:a2c5b5b8-f936-4c30-99f8-9cc04474d080 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:09.679426412 +0000 UTC m=+18.272700488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-25c9d" (UniqueName: "kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d") pod "network-check-target-5gknd" (UID: "a2c5b5b8-f936-4c30-99f8-9cc04474d080") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:20:01.912396 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:01.911896 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:20:01.912396 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:01.912041 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz99s" podUID="8a0fd497-acc1-4391-bb37-3a00c2c3362d" Jul 12 19:20:01.912396 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:01.912054 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:20:01.912396 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:01.912163 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gknd" podUID="a2c5b5b8-f936-4c30-99f8-9cc04474d080" Jul 12 19:20:01.912396 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:01.912190 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:20:01.912396 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:01.912253 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd2b6" podUID="697ae350-ebe5-4859-a7d4-389fd1da28f8" Jul 12 19:20:03.911285 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:03.911247 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:20:03.911710 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:03.911247 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:20:03.911710 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:03.911381 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gknd" podUID="a2c5b5b8-f936-4c30-99f8-9cc04474d080" Jul 12 19:20:03.911710 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:03.911253 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:20:03.911710 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:03.911486 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz99s" podUID="8a0fd497-acc1-4391-bb37-3a00c2c3362d" Jul 12 19:20:03.911710 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:03.911532 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd2b6" podUID="697ae350-ebe5-4859-a7d4-389fd1da28f8" Jul 12 19:20:05.910989 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:05.910956 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:20:05.911468 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:05.910956 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:20:05.911468 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:05.910956 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:20:05.911468 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:05.911142 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gknd" podUID="a2c5b5b8-f936-4c30-99f8-9cc04474d080" Jul 12 19:20:05.911468 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:05.911280 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd2b6" podUID="697ae350-ebe5-4859-a7d4-389fd1da28f8" Jul 12 19:20:05.911468 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:05.911369 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz99s" podUID="8a0fd497-acc1-4391-bb37-3a00c2c3362d" Jul 12 19:20:07.910555 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:07.910515 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:20:07.910951 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:07.910515 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:20:07.910951 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:07.910670 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz99s" podUID="8a0fd497-acc1-4391-bb37-3a00c2c3362d" Jul 12 19:20:07.910951 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:07.910515 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:20:07.910951 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:07.910835 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gknd" podUID="a2c5b5b8-f936-4c30-99f8-9cc04474d080" Jul 12 19:20:07.911254 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:07.910731 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd2b6" podUID="697ae350-ebe5-4859-a7d4-389fd1da28f8" Jul 12 19:20:09.644513 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:09.644477 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs\") pod \"network-metrics-daemon-fz99s\" (UID: \"8a0fd497-acc1-4391-bb37-3a00c2c3362d\") " pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:20:09.644971 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:09.644535 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret\") pod \"global-pull-secret-syncer-nd2b6\" (UID: \"697ae350-ebe5-4859-a7d4-389fd1da28f8\") " pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:20:09.644971 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:09.644629 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:20:09.644971 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:09.644641 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:20:09.644971 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:09.644681 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret podName:697ae350-ebe5-4859-a7d4-389fd1da28f8 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:25.644667391 +0000 UTC m=+34.237941467 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret") pod "global-pull-secret-syncer-nd2b6" (UID: "697ae350-ebe5-4859-a7d4-389fd1da28f8") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:20:09.644971 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:09.644710 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs podName:8a0fd497-acc1-4391-bb37-3a00c2c3362d nodeName:}" failed. No retries permitted until 2026-07-12 19:20:25.644690834 +0000 UTC m=+34.237964913 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs") pod "network-metrics-daemon-fz99s" (UID: "8a0fd497-acc1-4391-bb37-3a00c2c3362d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:20:09.745565 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:09.745524 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25c9d\" (UniqueName: \"kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d\") pod \"network-check-target-5gknd\" (UID: \"a2c5b5b8-f936-4c30-99f8-9cc04474d080\") " pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:20:09.745744 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:09.745711 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:20:09.745744 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:09.745736 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:20:09.745850 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:09.745751 2564 projected.go:196] Error preparing data for projected volume kube-api-access-25c9d for pod openshift-network-diagnostics/network-check-target-5gknd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:20:09.745850 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:09.745820 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d podName:a2c5b5b8-f936-4c30-99f8-9cc04474d080 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:25.745800771 +0000 UTC m=+34.339074847 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-25c9d" (UniqueName: "kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d") pod "network-check-target-5gknd" (UID: "a2c5b5b8-f936-4c30-99f8-9cc04474d080") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:20:09.910624 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:09.910583 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:20:09.910801 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:09.910603 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:20:09.910801 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:09.910726 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd2b6" podUID="697ae350-ebe5-4859-a7d4-389fd1da28f8" Jul 12 19:20:09.910801 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:09.910737 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:20:09.910971 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:09.910794 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gknd" podUID="a2c5b5b8-f936-4c30-99f8-9cc04474d080" Jul 12 19:20:09.910971 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:09.910875 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz99s" podUID="8a0fd497-acc1-4391-bb37-3a00c2c3362d" Jul 12 19:20:11.911046 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:11.910927 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:20:11.911046 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:11.911013 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:20:11.911046 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:11.911030 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gknd" podUID="a2c5b5b8-f936-4c30-99f8-9cc04474d080" Jul 12 19:20:11.911584 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:11.911178 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd2b6" podUID="697ae350-ebe5-4859-a7d4-389fd1da28f8" Jul 12 19:20:11.911584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:11.911198 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:20:11.911584 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:11.911341 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz99s" podUID="8a0fd497-acc1-4391-bb37-3a00c2c3362d" Jul 12 19:20:13.911247 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:13.911071 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:20:13.911977 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:13.911077 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:20:13.911977 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:13.911350 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gknd" podUID="a2c5b5b8-f936-4c30-99f8-9cc04474d080" Jul 12 19:20:13.911977 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:13.911494 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz99s" podUID="8a0fd497-acc1-4391-bb37-3a00c2c3362d" Jul 12 19:20:13.911977 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:13.911089 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:20:13.911977 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:13.911599 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd2b6" podUID="697ae350-ebe5-4859-a7d4-389fd1da28f8" Jul 12 19:20:14.027563 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:14.027527 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal" event={"ID":"72c5a76d2f244265b5a44f7b0562c46f","Type":"ContainerStarted","Data":"9266c282b392dce55f5142f1899882d88b54f94241d01edc2f0788898ab406c3"} Jul 12 19:20:14.028941 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:14.028913 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bsltc" event={"ID":"a4b25a35-4484-420e-8a26-dbfafbe0ac30","Type":"ContainerStarted","Data":"dca7037a2a03aab60b9fa90072e63b33a5240403748e328e93e19916b67a744d"} Jul 12 19:20:14.030239 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:14.030216 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-grbbl" event={"ID":"9a01d1fa-13e1-47af-b3af-2a9de7431dd1","Type":"ContainerStarted","Data":"c504ed753eb66d2d6d2bce4d48bf325df9591257a0d0a2cb3284f2474fc494ea"} Jul 12 19:20:14.031966 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:14.031942 2564 generic.go:356] "Generic (PLEG): container finished" podID="39bf2054-7b11-434a-a6e3-4d1f04cdd1a3" containerID="69d1b0a871d97be5a67b2f8e5f6491ae23a4501979e3ee4f87f6c8cdeffd6828" exitCode=0 Jul 12 19:20:14.032067 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:14.031975 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xv5j2" event={"ID":"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3","Type":"ContainerDied","Data":"69d1b0a871d97be5a67b2f8e5f6491ae23a4501979e3ee4f87f6c8cdeffd6828"} Jul 12 19:20:14.034558 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:14.034479 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" event={"ID":"0c1cc216-5111-43d9-979f-ad6cc1beece6","Type":"ContainerStarted","Data":"e28466b0db89142aac216a8c2ee97bc28567cd98272cd7f913044d7a0eaad0f1"} Jul 12 19:20:14.034650 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:14.034556 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" event={"ID":"0c1cc216-5111-43d9-979f-ad6cc1beece6","Type":"ContainerStarted","Data":"9bb778674055c71ca8f8c2836d374eb1c46a06b0cdc898cd72f1c9570ddb2443"} Jul 12 19:20:14.036052 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:14.035994 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fdm9d" event={"ID":"ea178531-de7c-44c8-911b-308836d78022","Type":"ContainerStarted","Data":"60424f9ef3acb074c6dfdfa758219011ed77623a3e8777790759afde417cd26e"} Jul 12 19:20:14.037288 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:14.037261 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-thbcj" event={"ID":"ab20a71c-5305-4984-8c11-e3b5e5451fcb","Type":"ContainerStarted","Data":"d8ef2953893098536232346281ca6f43e8a093078e39dc22f14932f65b7ce1e5"} Jul 12 19:20:14.038431 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:14.038403 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g4vrt" event={"ID":"a7a4e516-ce2a-4b83-8d99-83c61331242f","Type":"ContainerStarted","Data":"8f99f9c316cd56f8364fc78f2f727d574f35353b405b7395a67f556d7f8a4ae9"} Jul 12 19:20:14.043583 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:14.043546 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" event={"ID":"c0ce3d01-8fce-4587-9c82-f8673d917c05","Type":"ContainerStarted","Data":"b3910e977f375ff4296aa73efe17502e91d88b6a7731767977e552765bd00edd"} Jul 12 19:20:14.059655 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:14.059618 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-167.ec2.internal" podStartSLOduration=21.059603665 podStartE2EDuration="21.059603665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:14.043272303 +0000 UTC m=+22.636546403" watchObservedRunningTime="2026-07-12 19:20:14.059603665 +0000 UTC m=+22.652877741" Jul 12 19:20:14.060039 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:14.059987 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fdm9d" podStartSLOduration=3.644056002 podStartE2EDuration="22.059978064s" totalImagesPullingTime="18.415922062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:14.059654987 +0000 UTC m=+22.652929086" watchObservedRunningTime="2026-07-12 19:20:14.059978064 +0000 UTC m=+22.653252161" Jul 12 19:20:14.089421 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:14.089368 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-thbcj" podStartSLOduration=3.731898123 podStartE2EDuration="22.08934708s" totalImagesPullingTime="18.357448957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:14.089199245 +0000 UTC m=+22.682473342" watchObservedRunningTime="2026-07-12 19:20:14.08934708 +0000 UTC m=+22.682621192" Jul 12 19:20:14.102194 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:14.102150 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-grbbl" podStartSLOduration=8.368324003 podStartE2EDuration="22.102136649s" totalImagesPullingTime="13.733812646s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:14.101879094 +0000 UTC m=+22.695153191" watchObservedRunningTime="2026-07-12 19:20:14.102136649 +0000 UTC m=+22.695410747" Jul 12 19:20:14.122097 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:14.122062 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bsltc" podStartSLOduration=3.7778049 podStartE2EDuration="22.122048176s" totalImagesPullingTime="18.344243276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:14.122011865 +0000 UTC m=+22.715285963" watchObservedRunningTime="2026-07-12 19:20:14.122048176 +0000 UTC m=+22.715322271" Jul 12 19:20:14.135525 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:14.135490 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g4vrt" podStartSLOduration=3.773379968 podStartE2EDuration="22.135475129s" totalImagesPullingTime="18.362095161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:14.134942703 +0000 UTC m=+22.728216800" watchObservedRunningTime="2026-07-12 19:20:14.135475129 +0000 UTC m=+22.728749231" Jul 12 19:20:15.047362 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:15.047323 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l5rxz" event={"ID":"1a3e4a12-84ce-4021-8966-a0b8d3517fdc","Type":"ContainerStarted","Data":"f177ab098805bd1a317627e4070e295dc132467a1b1c940f1ec7f275d2adf083"} Jul 12 19:20:15.050826 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:15.050786 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" event={"ID":"0c1cc216-5111-43d9-979f-ad6cc1beece6","Type":"ContainerStarted","Data":"08addd887bf012cb098dc4f410ad6ce0d9fd15641358aebb25e2b1368ef7c200"} Jul 12 19:20:15.050975 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:15.050882 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" event={"ID":"0c1cc216-5111-43d9-979f-ad6cc1beece6","Type":"ContainerStarted","Data":"ea47106a2d0ba36bf8750e7c2583ff441fb159a59dcd95b1f18b7c7b8cfa4e88"} Jul 12 19:20:15.050975 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:15.050934 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" event={"ID":"0c1cc216-5111-43d9-979f-ad6cc1beece6","Type":"ContainerStarted","Data":"4bf85ffaa254aa56adb0e3639937c416b56f5565a9107133a05562cc4f2a400a"} Jul 12 19:20:15.051110 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:15.050981 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" event={"ID":"0c1cc216-5111-43d9-979f-ad6cc1beece6","Type":"ContainerStarted","Data":"5774bb65ca8a5f136a59105061420217f0f6909c6cc43a684b2ce6f310cf98eb"} Jul 12 19:20:15.214154 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:15.214118 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 12 19:20:15.757952 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:15.757920 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g4vrt" Jul 12 19:20:15.871062 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:15.870936 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-12T19:20:15.214140319Z","UUID":"f767dfe4-1469-4666-bac9-216bcf9a0e2c","Handler":null,"Name":"","Endpoint":""} Jul 12 19:20:15.873052 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:15.872985 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 12 19:20:15.873201 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:15.873062 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 12 19:20:15.911060 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:15.910907 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:20:15.911060 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:15.910909 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:20:15.911311 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:15.911167 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz99s" podUID="8a0fd497-acc1-4391-bb37-3a00c2c3362d" Jul 12 19:20:15.911311 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:15.911224 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:20:15.911311 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:15.911297 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd2b6" podUID="697ae350-ebe5-4859-a7d4-389fd1da28f8" Jul 12 19:20:15.911475 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:15.911447 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gknd" podUID="a2c5b5b8-f936-4c30-99f8-9cc04474d080" Jul 12 19:20:16.056316 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:16.056238 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" event={"ID":"c0ce3d01-8fce-4587-9c82-f8673d917c05","Type":"ContainerStarted","Data":"d91899cd093c3d35067644192acef1f0e73ed53fd046763a9c616c8bce8b7b46"} Jul 12 19:20:16.070725 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:16.070669 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l5rxz" podStartSLOduration=5.590324177 podStartE2EDuration="25.070650743s" totalImagesPullingTime="19.480326566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:19:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:16.07020595 +0000 UTC m=+24.663480050" watchObservedRunningTime="2026-07-12 19:20:16.070650743 +0000 UTC m=+24.663924842" Jul 12 19:20:16.846724 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:16.846191 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g4vrt" Jul 12 19:20:16.847009 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:16.846908 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g4vrt" Jul 12 19:20:17.061431 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:17.061390 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" event={"ID":"0c1cc216-5111-43d9-979f-ad6cc1beece6","Type":"ContainerStarted","Data":"e86bf7d687d173240e6f731e340f92af0618338273c62ee71268010bf41abcf4"} Jul 12 19:20:17.063398 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:17.063371 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" event={"ID":"c0ce3d01-8fce-4587-9c82-f8673d917c05","Type":"ContainerStarted","Data":"d0fab313b38eb94109217bf516f5ed0f9774a288d9e9f35326e3583722e0fe12"} Jul 12 19:20:17.066545 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:17.064469 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g4vrt" Jul 12 19:20:17.079473 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:17.079431 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr475" podStartSLOduration=8.865928028999999 podStartE2EDuration="25.079418135s" totalImagesPullingTime="16.213490106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:19:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:17.079110902 +0000 UTC m=+25.672385000" watchObservedRunningTime="2026-07-12 19:20:17.079418135 +0000 UTC m=+25.672692234" Jul 12 19:20:17.910426 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:17.910394 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:20:17.910645 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:17.910395 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:20:17.910645 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:17.910513 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz99s" podUID="8a0fd497-acc1-4391-bb37-3a00c2c3362d" Jul 12 19:20:17.910645 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:17.910618 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gknd" podUID="a2c5b5b8-f936-4c30-99f8-9cc04474d080" Jul 12 19:20:17.910812 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:17.910395 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:20:17.910812 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:17.910739 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd2b6" podUID="697ae350-ebe5-4859-a7d4-389fd1da28f8" Jul 12 19:20:19.072668 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:19.072319 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" event={"ID":"0c1cc216-5111-43d9-979f-ad6cc1beece6","Type":"ContainerStarted","Data":"90a90b0feb97eb84de543f2bb968f1830d08596e8cd26c33d20a9c0410a0091f"} Jul 12 19:20:19.072668 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:19.072606 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:20:19.088627 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:19.088591 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:20:19.097343 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:19.097277 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" podStartSLOduration=8.718006034 podStartE2EDuration="27.097260776s" totalImagesPullingTime="18.379254742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:19.096177865 +0000 UTC m=+27.689451976" watchObservedRunningTime="2026-07-12 19:20:19.097260776 +0000 UTC m=+27.690534868" Jul 12 19:20:19.911244 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:19.911209 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:20:19.911398 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:19.911214 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:20:19.911398 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:19.911326 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz99s" podUID="8a0fd497-acc1-4391-bb37-3a00c2c3362d" Jul 12 19:20:19.911398 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:19.911227 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:20:19.911500 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:19.911383 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gknd" podUID="a2c5b5b8-f936-4c30-99f8-9cc04474d080" Jul 12 19:20:19.911500 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:19.911484 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd2b6" podUID="697ae350-ebe5-4859-a7d4-389fd1da28f8" Jul 12 19:20:20.078563 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:20.078521 2564 generic.go:356] "Generic (PLEG): container finished" podID="39bf2054-7b11-434a-a6e3-4d1f04cdd1a3" containerID="95c0ae1fca717a8ea1edf9b6e6566f576fb9443dab6bf45f9228a70a41eb7891" exitCode=0 Jul 12 19:20:20.079036 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:20.078617 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xv5j2" event={"ID":"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3","Type":"ContainerDied","Data":"95c0ae1fca717a8ea1edf9b6e6566f576fb9443dab6bf45f9228a70a41eb7891"} Jul 12 19:20:20.079169 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:20.079151 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:20:20.079240 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:20.079228 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:20:20.095606 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:20.095587 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:20:20.918149 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:20.918115 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5gknd"] Jul 12 19:20:20.918406 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:20.918240 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:20:20.918406 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:20.918335 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gknd" podUID="a2c5b5b8-f936-4c30-99f8-9cc04474d080" Jul 12 19:20:20.921216 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:20.921159 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fz99s"] Jul 12 19:20:20.921347 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:20.921256 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:20:20.921347 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:20.921337 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz99s" podUID="8a0fd497-acc1-4391-bb37-3a00c2c3362d" Jul 12 19:20:20.921809 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:20.921786 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nd2b6"] Jul 12 19:20:20.921904 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:20.921872 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:20:20.921985 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:20.921961 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd2b6" podUID="697ae350-ebe5-4859-a7d4-389fd1da28f8" Jul 12 19:20:22.085326 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:22.085293 2564 generic.go:356] "Generic (PLEG): container finished" podID="39bf2054-7b11-434a-a6e3-4d1f04cdd1a3" containerID="e2be46d1d47ceea6c9bb1a7f6c73d9a11e69d67d9413f362fc660b3dbd03dbab" exitCode=0 Jul 12 19:20:22.085722 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:22.085390 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xv5j2" event={"ID":"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3","Type":"ContainerDied","Data":"e2be46d1d47ceea6c9bb1a7f6c73d9a11e69d67d9413f362fc660b3dbd03dbab"} Jul 12 19:20:22.910837 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:22.910655 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:20:22.911000 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:22.910655 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:20:22.911000 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:22.910911 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gknd" podUID="a2c5b5b8-f936-4c30-99f8-9cc04474d080" Jul 12 19:20:22.911000 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:22.910706 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:20:22.911000 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:22.910992 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz99s" podUID="8a0fd497-acc1-4391-bb37-3a00c2c3362d" Jul 12 19:20:22.911218 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:22.911113 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd2b6" podUID="697ae350-ebe5-4859-a7d4-389fd1da28f8" Jul 12 19:20:24.092242 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:24.092209 2564 generic.go:356] "Generic (PLEG): container finished" podID="39bf2054-7b11-434a-a6e3-4d1f04cdd1a3" containerID="3602155748f89e5a9dad7873d39d0c8417e57cfa35511c115120eb6529081930" exitCode=0 Jul 12 19:20:24.092814 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:24.092260 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xv5j2" event={"ID":"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3","Type":"ContainerDied","Data":"3602155748f89e5a9dad7873d39d0c8417e57cfa35511c115120eb6529081930"} Jul 12 19:20:24.910538 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:24.910498 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:20:24.910728 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:24.910498 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:20:24.910728 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:24.910618 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gknd" podUID="a2c5b5b8-f936-4c30-99f8-9cc04474d080" Jul 12 19:20:24.910728 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:24.910498 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:20:24.910904 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:24.910665 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd2b6" podUID="697ae350-ebe5-4859-a7d4-389fd1da28f8" Jul 12 19:20:24.910904 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:24.910744 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz99s" podUID="8a0fd497-acc1-4391-bb37-3a00c2c3362d" Jul 12 19:20:25.664538 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:25.664271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs\") pod \"network-metrics-daemon-fz99s\" (UID: \"8a0fd497-acc1-4391-bb37-3a00c2c3362d\") " pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:20:25.664538 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:25.664369 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret\") pod \"global-pull-secret-syncer-nd2b6\" (UID: \"697ae350-ebe5-4859-a7d4-389fd1da28f8\") " pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:20:25.664538 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:25.664430 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:20:25.664538 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:25.664461 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:20:25.664538 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:25.664501 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret podName:697ae350-ebe5-4859-a7d4-389fd1da28f8 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:57.664482885 +0000 UTC m=+66.257756961 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret") pod "global-pull-secret-syncer-nd2b6" (UID: "697ae350-ebe5-4859-a7d4-389fd1da28f8") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:20:25.664538 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:25.664519 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs podName:8a0fd497-acc1-4391-bb37-3a00c2c3362d nodeName:}" failed. No retries permitted until 2026-07-12 19:20:57.664510694 +0000 UTC m=+66.257784771 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs") pod "network-metrics-daemon-fz99s" (UID: "8a0fd497-acc1-4391-bb37-3a00c2c3362d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:20:25.765535 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:25.765496 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25c9d\" (UniqueName: \"kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d\") pod \"network-check-target-5gknd\" (UID: \"a2c5b5b8-f936-4c30-99f8-9cc04474d080\") " pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:20:25.765723 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:25.765686 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:20:25.765723 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:25.765715 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:20:25.765837 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:25.765728 2564 projected.go:196] Error preparing data for projected volume kube-api-access-25c9d for pod openshift-network-diagnostics/network-check-target-5gknd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:20:25.765837 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:25.765800 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d podName:a2c5b5b8-f936-4c30-99f8-9cc04474d080 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:57.765777988 +0000 UTC m=+66.359052081 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-25c9d" (UniqueName: "kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d") pod "network-check-target-5gknd" (UID: "a2c5b5b8-f936-4c30-99f8-9cc04474d080") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:20:26.229167 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.229142 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-167.ec2.internal" event="NodeReady" Jul 12 19:20:26.229313 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.229281 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 12 19:20:26.265353 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.265301 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7449988cff-n48gz"] Jul 12 19:20:26.267584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.267556 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-kvdtw"] Jul 12 19:20:26.267718 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.267674 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.269341 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.269318 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp"] Jul 12 19:20:26.269459 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.269442 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" Jul 12 19:20:26.270491 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.270470 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 12 19:20:26.270491 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.270482 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-29mst\"" Jul 12 19:20:26.270644 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.270470 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 12 19:20:26.270784 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.270766 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh"] Jul 12 19:20:26.270847 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.270803 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 12 19:20:26.270909 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.270872 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp" Jul 12 19:20:26.272187 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.272168 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-sdv2p\"" Jul 12 19:20:26.272283 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.272211 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 12 19:20:26.272624 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.272608 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k"] Jul 12 19:20:26.272727 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.272713 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" Jul 12 19:20:26.274910 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.274888 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:20:26.275011 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.274984 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 12 19:20:26.275336 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.274900 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 12 19:20:26.278230 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.278209 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 12 19:20:26.278643 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.278623 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 12 19:20:26.278757 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.278661 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 12 19:20:26.278757 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.278731 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 12 19:20:26.278885 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.278810 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-ctnjg\"" Jul 12 19:20:26.280745 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.280724 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 12 19:20:26.281005 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.280989 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 12 19:20:26.281298 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.281204 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 12 19:20:26.281298 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.281224 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 12 19:20:26.281926 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.281908 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 12 19:20:26.283054 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.282989 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp"] Jul 12 19:20:26.283917 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.283897 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9c99k"] Jul 12 19:20:26.287470 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.287449 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-kvdtw"] Jul 12 19:20:26.287557 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.287487 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zvbnn"] Jul 12 19:20:26.287597 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.287578 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9c99k" Jul 12 19:20:26.289365 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.289349 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7449988cff-n48gz"] Jul 12 19:20:26.289471 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.289374 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k"] Jul 12 19:20:26.289471 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.289435 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zvbnn" Jul 12 19:20:26.290818 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.290797 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 12 19:20:26.290818 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.290805 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 12 19:20:26.290970 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.290837 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-g7h2t\"" Jul 12 19:20:26.290970 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.290846 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 12 19:20:26.291605 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.291589 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 12 19:20:26.291798 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.291784 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qvb27\"" Jul 12 19:20:26.292007 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.291992 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 12 19:20:26.306909 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.306844 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9c99k"] Jul 12 19:20:26.307716 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.307692 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh"] Jul 12 19:20:26.308541 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.308514 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zvbnn"] Jul 12 19:20:26.369837 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.369810 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.369979 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.369849 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt7kv\" (UniqueName: \"kubernetes.io/projected/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-kube-api-access-rt7kv\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:20:26.369979 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.369887 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed03d0de-6326-40fa-afc9-639266c9b5f8-tmp\") pod \"klusterlet-addon-workmgr-86c47f4989-v7tnh\" (UID: \"ed03d0de-6326-40fa-afc9-639266c9b5f8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" Jul 12 19:20:26.369979 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.369911 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvnmk\" (UniqueName: \"kubernetes.io/projected/ed03d0de-6326-40fa-afc9-639266c9b5f8-kube-api-access-bvnmk\") pod \"klusterlet-addon-workmgr-86c47f4989-v7tnh\" (UID: \"ed03d0de-6326-40fa-afc9-639266c9b5f8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" Jul 12 19:20:26.369979 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.369953 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/418ab7c7-cad9-4e69-bde5-d16e643b12d7-trusted-ca\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.370181 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.369985 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/418ab7c7-cad9-4e69-bde5-d16e643b12d7-installation-pull-secrets\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.370181 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370035 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcbnt\" (UniqueName: \"kubernetes.io/projected/b6b3ac65-ccd5-4d7b-8809-8372731b54be-kube-api-access-lcbnt\") pod \"ingress-canary-9c99k\" (UID: \"b6b3ac65-ccd5-4d7b-8809-8372731b54be\") " pod="openshift-ingress-canary/ingress-canary-9c99k" Jul 12 19:20:26.370181 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370059 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6b3ac65-ccd5-4d7b-8809-8372731b54be-cert\") pod \"ingress-canary-9c99k\" (UID: \"b6b3ac65-ccd5-4d7b-8809-8372731b54be\") " pod="openshift-ingress-canary/ingress-canary-9c99k" Jul 12 19:20:26.370181 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370086 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:20:26.370181 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370110 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/47faa66e-e2bc-4670-9c86-a0d5ff7389d5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fb5b59cbd-fh49k\" (UID: \"47faa66e-e2bc-4670-9c86-a0d5ff7389d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:20:26.370181 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370135 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ed03d0de-6326-40fa-afc9-639266c9b5f8-klusterlet-config\") pod \"klusterlet-addon-workmgr-86c47f4989-v7tnh\" (UID: \"ed03d0de-6326-40fa-afc9-639266c9b5f8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" Jul 12 19:20:26.370512 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370240 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-certificates\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.370512 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370290 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/418ab7c7-cad9-4e69-bde5-d16e643b12d7-ca-trust-extracted\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.370512 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370368 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/47faa66e-e2bc-4670-9c86-a0d5ff7389d5-hub\") pod \"cluster-proxy-proxy-agent-6fb5b59cbd-fh49k\" (UID: \"47faa66e-e2bc-4670-9c86-a0d5ff7389d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:20:26.370512 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370396 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hhqp\" (UniqueName: \"kubernetes.io/projected/2da3d858-73b3-4a08-8678-01155c9115e7-kube-api-access-5hhqp\") pod \"managed-serviceaccount-addon-agent-7496b85cbd-t5xcp\" (UID: \"2da3d858-73b3-4a08-8678-01155c9115e7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp" Jul 12 19:20:26.370512 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370428 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkn2w\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-kube-api-access-wkn2w\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.370512 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370456 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/47faa66e-e2bc-4670-9c86-a0d5ff7389d5-ca\") pod \"cluster-proxy-proxy-agent-6fb5b59cbd-fh49k\" (UID: \"47faa66e-e2bc-4670-9c86-a0d5ff7389d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:20:26.370512 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370497 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-kvdtw\" (UID: \"5efb57d2-5bf0-4799-8e22-3a3066c94cb2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" Jul 12 19:20:26.370863 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370527 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6lsk\" (UniqueName: \"kubernetes.io/projected/47faa66e-e2bc-4670-9c86-a0d5ff7389d5-kube-api-access-t6lsk\") pod \"cluster-proxy-proxy-agent-6fb5b59cbd-fh49k\" (UID: \"47faa66e-e2bc-4670-9c86-a0d5ff7389d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:20:26.370863 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370552 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2da3d858-73b3-4a08-8678-01155c9115e7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7496b85cbd-t5xcp\" (UID: \"2da3d858-73b3-4a08-8678-01155c9115e7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp" Jul 12 19:20:26.370863 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370604 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47faa66e-e2bc-4670-9c86-a0d5ff7389d5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fb5b59cbd-fh49k\" (UID: \"47faa66e-e2bc-4670-9c86-a0d5ff7389d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:20:26.370863 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370647 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/47faa66e-e2bc-4670-9c86-a0d5ff7389d5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fb5b59cbd-fh49k\" (UID: \"47faa66e-e2bc-4670-9c86-a0d5ff7389d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:20:26.370863 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370670 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-tmp-dir\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:20:26.370863 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-nginx-conf\") pod \"networking-console-plugin-6cf856758c-kvdtw\" (UID: \"5efb57d2-5bf0-4799-8e22-3a3066c94cb2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" Jul 12 19:20:26.370863 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370726 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-config-volume\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:20:26.370863 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370768 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/418ab7c7-cad9-4e69-bde5-d16e643b12d7-image-registry-private-configuration\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.370863 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.370796 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-bound-sa-token\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.451403 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.451365 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:26.471890 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.471850 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/418ab7c7-cad9-4e69-bde5-d16e643b12d7-ca-trust-extracted\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.471890 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.471889 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/47faa66e-e2bc-4670-9c86-a0d5ff7389d5-hub\") pod \"cluster-proxy-proxy-agent-6fb5b59cbd-fh49k\" (UID: \"47faa66e-e2bc-4670-9c86-a0d5ff7389d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:20:26.472143 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.471907 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hhqp\" (UniqueName: \"kubernetes.io/projected/2da3d858-73b3-4a08-8678-01155c9115e7-kube-api-access-5hhqp\") pod \"managed-serviceaccount-addon-agent-7496b85cbd-t5xcp\" (UID: \"2da3d858-73b3-4a08-8678-01155c9115e7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp" Jul 12 19:20:26.472143 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.471923 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkn2w\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-kube-api-access-wkn2w\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.472143 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.471945 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/47faa66e-e2bc-4670-9c86-a0d5ff7389d5-ca\") pod \"cluster-proxy-proxy-agent-6fb5b59cbd-fh49k\" (UID: \"47faa66e-e2bc-4670-9c86-a0d5ff7389d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:20:26.472143 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.471972 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-kvdtw\" (UID: \"5efb57d2-5bf0-4799-8e22-3a3066c94cb2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" Jul 12 19:20:26.472143 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472006 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6lsk\" (UniqueName: \"kubernetes.io/projected/47faa66e-e2bc-4670-9c86-a0d5ff7389d5-kube-api-access-t6lsk\") pod \"cluster-proxy-proxy-agent-6fb5b59cbd-fh49k\" (UID: \"47faa66e-e2bc-4670-9c86-a0d5ff7389d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:20:26.472143 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472050 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2da3d858-73b3-4a08-8678-01155c9115e7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7496b85cbd-t5xcp\" (UID: \"2da3d858-73b3-4a08-8678-01155c9115e7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp" Jul 12 19:20:26.472143 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472085 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47faa66e-e2bc-4670-9c86-a0d5ff7389d5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fb5b59cbd-fh49k\" (UID: \"47faa66e-e2bc-4670-9c86-a0d5ff7389d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:20:26.472143 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472124 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/47faa66e-e2bc-4670-9c86-a0d5ff7389d5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fb5b59cbd-fh49k\" (UID: \"47faa66e-e2bc-4670-9c86-a0d5ff7389d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:20:26.472143 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:26.472133 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 12 19:20:26.472143 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472147 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-tmp-dir\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:20:26.472613 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472178 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-nginx-conf\") pod \"networking-console-plugin-6cf856758c-kvdtw\" (UID: \"5efb57d2-5bf0-4799-8e22-3a3066c94cb2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" Jul 12 19:20:26.472613 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472201 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-config-volume\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:20:26.472613 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472230 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/418ab7c7-cad9-4e69-bde5-d16e643b12d7-image-registry-private-configuration\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.472613 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472257 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-bound-sa-token\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.472613 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472296 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.472613 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472332 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/418ab7c7-cad9-4e69-bde5-d16e643b12d7-ca-trust-extracted\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.472613 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472321 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt7kv\" (UniqueName: \"kubernetes.io/projected/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-kube-api-access-rt7kv\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:20:26.472613 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:26.472388 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert podName:5efb57d2-5bf0-4799-8e22-3a3066c94cb2 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:26.972369188 +0000 UTC m=+35.565643265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-kvdtw" (UID: "5efb57d2-5bf0-4799-8e22-3a3066c94cb2") : secret "networking-console-plugin-cert" not found Jul 12 19:20:26.472613 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472420 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed03d0de-6326-40fa-afc9-639266c9b5f8-tmp\") pod \"klusterlet-addon-workmgr-86c47f4989-v7tnh\" (UID: \"ed03d0de-6326-40fa-afc9-639266c9b5f8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" Jul 12 19:20:26.472613 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472445 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvnmk\" (UniqueName: \"kubernetes.io/projected/ed03d0de-6326-40fa-afc9-639266c9b5f8-kube-api-access-bvnmk\") pod \"klusterlet-addon-workmgr-86c47f4989-v7tnh\" (UID: \"ed03d0de-6326-40fa-afc9-639266c9b5f8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" Jul 12 19:20:26.472613 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/418ab7c7-cad9-4e69-bde5-d16e643b12d7-trusted-ca\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.472613 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472490 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/418ab7c7-cad9-4e69-bde5-d16e643b12d7-installation-pull-secrets\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.472613 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472548 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcbnt\" (UniqueName: \"kubernetes.io/projected/b6b3ac65-ccd5-4d7b-8809-8372731b54be-kube-api-access-lcbnt\") pod \"ingress-canary-9c99k\" (UID: \"b6b3ac65-ccd5-4d7b-8809-8372731b54be\") " pod="openshift-ingress-canary/ingress-canary-9c99k" Jul 12 19:20:26.472613 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472571 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6b3ac65-ccd5-4d7b-8809-8372731b54be-cert\") pod \"ingress-canary-9c99k\" (UID: \"b6b3ac65-ccd5-4d7b-8809-8372731b54be\") " pod="openshift-ingress-canary/ingress-canary-9c99k" Jul 12 19:20:26.472613 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472600 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:20:26.475080 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472639 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/47faa66e-e2bc-4670-9c86-a0d5ff7389d5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fb5b59cbd-fh49k\" (UID: \"47faa66e-e2bc-4670-9c86-a0d5ff7389d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:20:26.475080 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472663 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ed03d0de-6326-40fa-afc9-639266c9b5f8-klusterlet-config\") pod \"klusterlet-addon-workmgr-86c47f4989-v7tnh\" (UID: \"ed03d0de-6326-40fa-afc9-639266c9b5f8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" Jul 12 19:20:26.475080 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.472693 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-certificates\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.475080 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.473117 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-config-volume\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:20:26.475080 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.473145 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/47faa66e-e2bc-4670-9c86-a0d5ff7389d5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fb5b59cbd-fh49k\" (UID: \"47faa66e-e2bc-4670-9c86-a0d5ff7389d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:20:26.475080 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:26.473227 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 12 19:20:26.475080 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:26.473253 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7449988cff-n48gz: secret "image-registry-tls" not found Jul 12 19:20:26.475080 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.473279 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-certificates\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.475080 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:26.473307 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls podName:418ab7c7-cad9-4e69-bde5-d16e643b12d7 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:26.973289903 +0000 UTC m=+35.566563992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls") pod "image-registry-7449988cff-n48gz" (UID: "418ab7c7-cad9-4e69-bde5-d16e643b12d7") : secret "image-registry-tls" not found Jul 12 19:20:26.475080 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.473348 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-tmp-dir\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:20:26.475080 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.473550 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed03d0de-6326-40fa-afc9-639266c9b5f8-tmp\") pod \"klusterlet-addon-workmgr-86c47f4989-v7tnh\" (UID: \"ed03d0de-6326-40fa-afc9-639266c9b5f8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" Jul 12 19:20:26.475080 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:26.473571 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:20:26.475080 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:26.473614 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls podName:174905d8-e6cd-4e9b-a6fa-02d6f96d05d5 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:26.973600326 +0000 UTC m=+35.566874405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls") pod "dns-default-zvbnn" (UID: "174905d8-e6cd-4e9b-a6fa-02d6f96d05d5") : secret "dns-default-metrics-tls" not found Jul 12 19:20:26.475080 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.474723 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/418ab7c7-cad9-4e69-bde5-d16e643b12d7-trusted-ca\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.475080 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.475104 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-nginx-conf\") pod \"networking-console-plugin-6cf856758c-kvdtw\" (UID: \"5efb57d2-5bf0-4799-8e22-3a3066c94cb2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" Jul 12 19:20:26.477965 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.477944 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/47faa66e-e2bc-4670-9c86-a0d5ff7389d5-hub\") pod \"cluster-proxy-proxy-agent-6fb5b59cbd-fh49k\" (UID: \"47faa66e-e2bc-4670-9c86-a0d5ff7389d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:20:26.478074 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.478047 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2da3d858-73b3-4a08-8678-01155c9115e7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7496b85cbd-t5xcp\" (UID: \"2da3d858-73b3-4a08-8678-01155c9115e7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp" Jul 12 19:20:26.478148 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.478122 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ed03d0de-6326-40fa-afc9-639266c9b5f8-klusterlet-config\") pod \"klusterlet-addon-workmgr-86c47f4989-v7tnh\" (UID: \"ed03d0de-6326-40fa-afc9-639266c9b5f8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" Jul 12 19:20:26.478337 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.478321 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/47faa66e-e2bc-4670-9c86-a0d5ff7389d5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fb5b59cbd-fh49k\" (UID: \"47faa66e-e2bc-4670-9c86-a0d5ff7389d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:20:26.479484 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.479455 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/418ab7c7-cad9-4e69-bde5-d16e643b12d7-installation-pull-secrets\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.479714 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.479628 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/47faa66e-e2bc-4670-9c86-a0d5ff7389d5-ca\") pod \"cluster-proxy-proxy-agent-6fb5b59cbd-fh49k\" (UID: \"47faa66e-e2bc-4670-9c86-a0d5ff7389d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:20:26.479862 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.479841 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/418ab7c7-cad9-4e69-bde5-d16e643b12d7-image-registry-private-configuration\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.479957 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.479862 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47faa66e-e2bc-4670-9c86-a0d5ff7389d5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fb5b59cbd-fh49k\" (UID: \"47faa66e-e2bc-4670-9c86-a0d5ff7389d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:20:26.479957 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.479936 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6b3ac65-ccd5-4d7b-8809-8372731b54be-cert\") pod \"ingress-canary-9c99k\" (UID: \"b6b3ac65-ccd5-4d7b-8809-8372731b54be\") " pod="openshift-ingress-canary/ingress-canary-9c99k" Jul 12 19:20:26.485657 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.485610 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkn2w\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-kube-api-access-wkn2w\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.486375 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.485793 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-bound-sa-token\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.487094 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.486993 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hhqp\" (UniqueName: \"kubernetes.io/projected/2da3d858-73b3-4a08-8678-01155c9115e7-kube-api-access-5hhqp\") pod \"managed-serviceaccount-addon-agent-7496b85cbd-t5xcp\" (UID: \"2da3d858-73b3-4a08-8678-01155c9115e7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp" Jul 12 19:20:26.487674 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.487652 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6lsk\" (UniqueName: \"kubernetes.io/projected/47faa66e-e2bc-4670-9c86-a0d5ff7389d5-kube-api-access-t6lsk\") pod \"cluster-proxy-proxy-agent-6fb5b59cbd-fh49k\" (UID: \"47faa66e-e2bc-4670-9c86-a0d5ff7389d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:20:26.488127 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.488007 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt7kv\" (UniqueName: \"kubernetes.io/projected/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-kube-api-access-rt7kv\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:20:26.488355 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.488336 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvnmk\" (UniqueName: \"kubernetes.io/projected/ed03d0de-6326-40fa-afc9-639266c9b5f8-kube-api-access-bvnmk\") pod \"klusterlet-addon-workmgr-86c47f4989-v7tnh\" (UID: \"ed03d0de-6326-40fa-afc9-639266c9b5f8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" Jul 12 19:20:26.489012 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.488992 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcbnt\" (UniqueName: \"kubernetes.io/projected/b6b3ac65-ccd5-4d7b-8809-8372731b54be-kube-api-access-lcbnt\") pod \"ingress-canary-9c99k\" (UID: \"b6b3ac65-ccd5-4d7b-8809-8372731b54be\") " pod="openshift-ingress-canary/ingress-canary-9c99k" Jul 12 19:20:26.533167 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.533126 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:26.610652 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.610618 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp" Jul 12 19:20:26.618084 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.618062 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" Jul 12 19:20:26.625449 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.625328 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:20:26.645940 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.645848 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9c99k" Jul 12 19:20:26.825769 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.825527 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:26.829792 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.829763 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh"] Jul 12 19:20:26.833634 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:20:26.833609 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded03d0de_6326_40fa_afc9_639266c9b5f8.slice/crio-08210133ed0f9b05ac7a27f071d4db695fc103e9786c582d22ba187cf0c86867 WatchSource:0}: Error finding container 08210133ed0f9b05ac7a27f071d4db695fc103e9786c582d22ba187cf0c86867: Status 404 returned error can't find the container with id 08210133ed0f9b05ac7a27f071d4db695fc103e9786c582d22ba187cf0c86867 Jul 12 19:20:26.835270 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.835243 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp"] Jul 12 19:20:26.838425 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:20:26.838397 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2da3d858_73b3_4a08_8678_01155c9115e7.slice/crio-60474c35c1829117f53f98f4fb717fbf40fe316bf58457746e28ffa7429b573c WatchSource:0}: Error finding container 60474c35c1829117f53f98f4fb717fbf40fe316bf58457746e28ffa7429b573c: Status 404 returned error can't find the container with id 60474c35c1829117f53f98f4fb717fbf40fe316bf58457746e28ffa7429b573c Jul 12 19:20:26.843867 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.843847 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k"] Jul 12 19:20:26.846711 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.846690 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9c99k"] Jul 12 19:20:26.847180 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:20:26.847159 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47faa66e_e2bc_4670_9c86_a0d5ff7389d5.slice/crio-4590bc038758f5569f90a40171394b4a06af825eacc9332d01c8c993dd5cf6e6 WatchSource:0}: Error finding container 4590bc038758f5569f90a40171394b4a06af825eacc9332d01c8c993dd5cf6e6: Status 404 returned error can't find the container with id 4590bc038758f5569f90a40171394b4a06af825eacc9332d01c8c993dd5cf6e6 Jul 12 19:20:26.850513 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:20:26.850491 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b3ac65_ccd5_4d7b_8809_8372731b54be.slice/crio-a07d50956309cc269b0c6ac2b4a562061bddba035c4dcf85484468173211f88e WatchSource:0}: Error finding container a07d50956309cc269b0c6ac2b4a562061bddba035c4dcf85484468173211f88e: Status 404 returned error can't find the container with id a07d50956309cc269b0c6ac2b4a562061bddba035c4dcf85484468173211f88e Jul 12 19:20:26.910661 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.910629 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:20:26.910800 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.910674 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:20:26.910800 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.910733 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:20:26.913675 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.913654 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 12 19:20:26.913799 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.913716 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zkgdj\"" Jul 12 19:20:26.913799 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.913747 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 12 19:20:26.914001 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.913983 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 12 19:20:26.914001 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.913997 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 12 19:20:26.914146 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.914000 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-brd5c\"" Jul 12 19:20:26.919717 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.919697 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:26.977086 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.977040 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:26.977277 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.977107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:20:26.977277 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:26.977157 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-kvdtw\" (UID: \"5efb57d2-5bf0-4799-8e22-3a3066c94cb2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" Jul 12 19:20:26.977277 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:26.977200 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 12 19:20:26.977277 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:26.977218 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7449988cff-n48gz: secret "image-registry-tls" not found Jul 12 19:20:26.977277 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:26.977262 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 12 19:20:26.977277 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:26.977265 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:20:26.977493 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:26.977276 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls podName:418ab7c7-cad9-4e69-bde5-d16e643b12d7 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:27.977259509 +0000 UTC m=+36.570533598 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls") pod "image-registry-7449988cff-n48gz" (UID: "418ab7c7-cad9-4e69-bde5-d16e643b12d7") : secret "image-registry-tls" not found Jul 12 19:20:26.977493 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:26.977334 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert podName:5efb57d2-5bf0-4799-8e22-3a3066c94cb2 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:27.977315 +0000 UTC m=+36.570589078 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-kvdtw" (UID: "5efb57d2-5bf0-4799-8e22-3a3066c94cb2") : secret "networking-console-plugin-cert" not found Jul 12 19:20:26.977493 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:26.977350 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls podName:174905d8-e6cd-4e9b-a6fa-02d6f96d05d5 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:27.977339557 +0000 UTC m=+36.570613637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls") pod "dns-default-zvbnn" (UID: "174905d8-e6cd-4e9b-a6fa-02d6f96d05d5") : secret "dns-default-metrics-tls" not found Jul 12 19:20:27.100876 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:27.100836 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" event={"ID":"ed03d0de-6326-40fa-afc9-639266c9b5f8","Type":"ContainerStarted","Data":"08210133ed0f9b05ac7a27f071d4db695fc103e9786c582d22ba187cf0c86867"} Jul 12 19:20:27.102193 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:27.102160 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp" event={"ID":"2da3d858-73b3-4a08-8678-01155c9115e7","Type":"ContainerStarted","Data":"60474c35c1829117f53f98f4fb717fbf40fe316bf58457746e28ffa7429b573c"} Jul 12 19:20:27.103400 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:27.103373 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" event={"ID":"47faa66e-e2bc-4670-9c86-a0d5ff7389d5","Type":"ContainerStarted","Data":"4590bc038758f5569f90a40171394b4a06af825eacc9332d01c8c993dd5cf6e6"} Jul 12 19:20:27.104541 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:27.104513 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9c99k" event={"ID":"b6b3ac65-ccd5-4d7b-8809-8372731b54be","Type":"ContainerStarted","Data":"a07d50956309cc269b0c6ac2b4a562061bddba035c4dcf85484468173211f88e"} Jul 12 19:20:27.989370 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:27.988750 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:27.989370 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:27.988808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:20:27.989370 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:27.988846 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-kvdtw\" (UID: \"5efb57d2-5bf0-4799-8e22-3a3066c94cb2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" Jul 12 19:20:27.989370 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:27.988965 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 12 19:20:27.989370 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:27.989013 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert podName:5efb57d2-5bf0-4799-8e22-3a3066c94cb2 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:29.989000006 +0000 UTC m=+38.582274083 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-kvdtw" (UID: "5efb57d2-5bf0-4799-8e22-3a3066c94cb2") : secret "networking-console-plugin-cert" not found Jul 12 19:20:27.989370 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:27.989147 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:20:27.989370 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:27.989187 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 12 19:20:27.989370 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:27.989211 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7449988cff-n48gz: secret "image-registry-tls" not found Jul 12 19:20:27.989370 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:27.989192 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls podName:174905d8-e6cd-4e9b-a6fa-02d6f96d05d5 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:29.989178968 +0000 UTC m=+38.582453047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls") pod "dns-default-zvbnn" (UID: "174905d8-e6cd-4e9b-a6fa-02d6f96d05d5") : secret "dns-default-metrics-tls" not found Jul 12 19:20:27.989370 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:27.989281 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls podName:418ab7c7-cad9-4e69-bde5-d16e643b12d7 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:29.989264102 +0000 UTC m=+38.582538184 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls") pod "image-registry-7449988cff-n48gz" (UID: "418ab7c7-cad9-4e69-bde5-d16e643b12d7") : secret "image-registry-tls" not found Jul 12 19:20:28.236774 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:28.236676 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:28.360418 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:28.360281 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:28.912737 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:28.912697 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:29.015241 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:29.015138 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:30.007794 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:30.007755 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:20:30.007997 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:30.007843 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-kvdtw\" (UID: \"5efb57d2-5bf0-4799-8e22-3a3066c94cb2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" Jul 12 19:20:30.007997 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:30.007935 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:30.009048 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:30.008102 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 12 19:20:30.009048 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:30.008443 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7449988cff-n48gz: secret "image-registry-tls" not found Jul 12 19:20:30.009048 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:30.008539 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls podName:418ab7c7-cad9-4e69-bde5-d16e643b12d7 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:34.008505273 +0000 UTC m=+42.601779362 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls") pod "image-registry-7449988cff-n48gz" (UID: "418ab7c7-cad9-4e69-bde5-d16e643b12d7") : secret "image-registry-tls" not found Jul 12 19:20:30.011511 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:30.011489 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:20:30.011616 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:30.011594 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls podName:174905d8-e6cd-4e9b-a6fa-02d6f96d05d5 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:34.011572789 +0000 UTC m=+42.604846888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls") pod "dns-default-zvbnn" (UID: "174905d8-e6cd-4e9b-a6fa-02d6f96d05d5") : secret "dns-default-metrics-tls" not found Jul 12 19:20:30.011616 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:30.011490 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 12 19:20:30.011737 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:30.011675 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert podName:5efb57d2-5bf0-4799-8e22-3a3066c94cb2 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:34.01166244 +0000 UTC m=+42.604936531 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-kvdtw" (UID: "5efb57d2-5bf0-4799-8e22-3a3066c94cb2") : secret "networking-console-plugin-cert" not found Jul 12 19:20:32.256050 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:32.255998 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:32.356186 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:32.356147 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:34.048541 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:34.048508 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:20:34.048981 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:34.048566 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-kvdtw\" (UID: \"5efb57d2-5bf0-4799-8e22-3a3066c94cb2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" Jul 12 19:20:34.048981 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:34.048677 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:34.048981 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:34.048691 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 12 19:20:34.048981 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:34.048692 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:20:34.048981 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:34.048764 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert podName:5efb57d2-5bf0-4799-8e22-3a3066c94cb2 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:42.048741931 +0000 UTC m=+50.642016029 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-kvdtw" (UID: "5efb57d2-5bf0-4799-8e22-3a3066c94cb2") : secret "networking-console-plugin-cert" not found Jul 12 19:20:34.048981 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:34.048806 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 12 19:20:34.048981 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:34.048817 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7449988cff-n48gz: secret "image-registry-tls" not found Jul 12 19:20:34.048981 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:34.048817 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls podName:174905d8-e6cd-4e9b-a6fa-02d6f96d05d5 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:42.048794836 +0000 UTC m=+50.642068916 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls") pod "dns-default-zvbnn" (UID: "174905d8-e6cd-4e9b-a6fa-02d6f96d05d5") : secret "dns-default-metrics-tls" not found Jul 12 19:20:34.048981 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:34.048863 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls podName:418ab7c7-cad9-4e69-bde5-d16e643b12d7 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:42.048850992 +0000 UTC m=+50.642125068 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls") pod "image-registry-7449988cff-n48gz" (UID: "418ab7c7-cad9-4e69-bde5-d16e643b12d7") : secret "image-registry-tls" not found Jul 12 19:20:37.145617 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:37.145528 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp" event={"ID":"2da3d858-73b3-4a08-8678-01155c9115e7","Type":"ContainerStarted","Data":"bdcf6e5c0edfa75e01746737560de840d17d169b5be729bce026c4bc6f1a3fad"} Jul 12 19:20:37.148235 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:37.148211 2564 generic.go:356] "Generic (PLEG): container finished" podID="39bf2054-7b11-434a-a6e3-4d1f04cdd1a3" containerID="c44f7f670b0ccfc6b1edfe86d5bcbdda14864cfe51179df1ae293abd0e5c98a3" exitCode=0 Jul 12 19:20:37.148365 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:37.148289 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xv5j2" event={"ID":"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3","Type":"ContainerDied","Data":"c44f7f670b0ccfc6b1edfe86d5bcbdda14864cfe51179df1ae293abd0e5c98a3"} Jul 12 19:20:37.150455 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:37.150432 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" event={"ID":"47faa66e-e2bc-4670-9c86-a0d5ff7389d5","Type":"ContainerStarted","Data":"8ee04ff35aa905a5a45fbb285270654cf44a9f541db19506f1c77729f8da394a"} Jul 12 19:20:37.150576 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:37.150469 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" event={"ID":"47faa66e-e2bc-4670-9c86-a0d5ff7389d5","Type":"ContainerStarted","Data":"f63e34e951791dabe7562a7023648c750bcf81322759b9058f8738c64ce9aacb"} Jul 12 19:20:37.150576 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:37.150485 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" event={"ID":"47faa66e-e2bc-4670-9c86-a0d5ff7389d5","Type":"ContainerStarted","Data":"1f36c6b61e31bff1e5dd82fa7db022da7f7b960d6e9e684f46d055b95b583451"} Jul 12 19:20:37.151866 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:37.151816 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9c99k" event={"ID":"b6b3ac65-ccd5-4d7b-8809-8372731b54be","Type":"ContainerStarted","Data":"367314055b29bf59779b87fd3b707434fa9027f0044810eaa0f434fd0291988c"} Jul 12 19:20:37.153264 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:37.153243 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" event={"ID":"ed03d0de-6326-40fa-afc9-639266c9b5f8","Type":"ContainerStarted","Data":"204b98032bd924a5b28e8520b26e42e1d5ca0dbf40774ca432f3b7555f3a5fde"} Jul 12 19:20:37.153413 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:37.153401 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" Jul 12 19:20:37.155122 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:37.155106 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" Jul 12 19:20:37.160437 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:37.160397 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp" podStartSLOduration=15.544120606 podStartE2EDuration="25.160385412s" totalImagesPullingTime="9.616264806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:37.159616011 +0000 UTC m=+45.752890110" watchObservedRunningTime="2026-07-12 19:20:37.160385412 +0000 UTC m=+45.753659517" Jul 12 19:20:37.178127 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:37.178086 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" podStartSLOduration=15.570050491 podStartE2EDuration="25.178072502s" totalImagesPullingTime="9.608022011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:37.176969923 +0000 UTC m=+45.770244025" watchObservedRunningTime="2026-07-12 19:20:37.178072502 +0000 UTC m=+45.771346599" Jul 12 19:20:37.194918 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:37.194863 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" podStartSLOduration=15.557216869 podStartE2EDuration="25.19484594s" totalImagesPullingTime="9.637629071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:37.191468705 +0000 UTC m=+45.784742803" watchObservedRunningTime="2026-07-12 19:20:37.19484594 +0000 UTC m=+45.788120038" Jul 12 19:20:37.228282 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:37.228236 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9c99k" podStartSLOduration=1.623397063 podStartE2EDuration="11.228221528s" totalImagesPullingTime="9.604824465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:20:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:37.227380367 +0000 UTC m=+45.820654465" watchObservedRunningTime="2026-07-12 19:20:37.228221528 +0000 UTC m=+45.821495687" Jul 12 19:20:38.158657 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:38.158619 2564 generic.go:356] "Generic (PLEG): container finished" podID="39bf2054-7b11-434a-a6e3-4d1f04cdd1a3" containerID="eac69ab299a289c8946cf766a8317ada934aa81880ac70c006996423e3a795d1" exitCode=0 Jul 12 19:20:38.159106 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:38.158734 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xv5j2" event={"ID":"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3","Type":"ContainerDied","Data":"eac69ab299a289c8946cf766a8317ada934aa81880ac70c006996423e3a795d1"} Jul 12 19:20:39.165286 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:39.165244 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xv5j2" event={"ID":"39bf2054-7b11-434a-a6e3-4d1f04cdd1a3","Type":"ContainerStarted","Data":"2457c31d93999130ee72032e5858272fca00ce9f973b45380f1aa76cb7f43819"} Jul 12 19:20:39.188535 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:39.188485 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xv5j2" podStartSLOduration=9.130259631 podStartE2EDuration="48.188469196s" totalImagesPullingTime="39.058209565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:19:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:39.186550544 +0000 UTC m=+47.779824644" watchObservedRunningTime="2026-07-12 19:20:39.188469196 +0000 UTC m=+47.781743296" Jul 12 19:20:39.307424 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:39.307392 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:39.407933 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:39.407880 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:40.392433 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:40.392391 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:40.496015 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:40.495984 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:40.551091 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:40.551057 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:40.643307 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:40.643221 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:40.743490 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:40.743455 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:40.956882 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:40.956850 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:41.006543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:41.006510 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:41.338058 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:41.337956 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:41.385746 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:41.385718 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:42.110944 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:42.110902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-kvdtw\" (UID: \"5efb57d2-5bf0-4799-8e22-3a3066c94cb2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" Jul 12 19:20:42.111419 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:42.110988 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:42.111419 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:42.111047 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:20:42.111419 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:42.111102 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 12 19:20:42.111419 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:42.111144 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:20:42.111419 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:42.111148 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 12 19:20:42.111419 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:42.111165 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7449988cff-n48gz: secret "image-registry-tls" not found Jul 12 19:20:42.111419 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:42.111190 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert podName:5efb57d2-5bf0-4799-8e22-3a3066c94cb2 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:58.111170339 +0000 UTC m=+66.704444415 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-kvdtw" (UID: "5efb57d2-5bf0-4799-8e22-3a3066c94cb2") : secret "networking-console-plugin-cert" not found Jul 12 19:20:42.111419 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:42.111212 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls podName:174905d8-e6cd-4e9b-a6fa-02d6f96d05d5 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:58.111202416 +0000 UTC m=+66.704476500 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls") pod "dns-default-zvbnn" (UID: "174905d8-e6cd-4e9b-a6fa-02d6f96d05d5") : secret "dns-default-metrics-tls" not found Jul 12 19:20:42.111419 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:42.111229 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls podName:418ab7c7-cad9-4e69-bde5-d16e643b12d7 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:58.11122028 +0000 UTC m=+66.704494358 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls") pod "image-registry-7449988cff-n48gz" (UID: "418ab7c7-cad9-4e69-bde5-d16e643b12d7") : secret "image-registry-tls" not found Jul 12 19:20:52.104690 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:52.104661 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jj2hs" Jul 12 19:20:52.254772 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:52.254736 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:52.350030 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:52.349983 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:20:57.740816 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:57.740769 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs\") pod \"network-metrics-daemon-fz99s\" (UID: \"8a0fd497-acc1-4391-bb37-3a00c2c3362d\") " pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:20:57.741296 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:57.740833 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret\") pod \"global-pull-secret-syncer-nd2b6\" (UID: \"697ae350-ebe5-4859-a7d4-389fd1da28f8\") " pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:20:57.743321 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:57.743301 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 12 19:20:57.743392 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:57.743346 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 12 19:20:57.751229 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:57.751205 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 12 19:20:57.751341 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:57.751280 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs podName:8a0fd497-acc1-4391-bb37-3a00c2c3362d nodeName:}" failed. No retries permitted until 2026-07-12 19:22:01.751262373 +0000 UTC m=+130.344536451 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs") pod "network-metrics-daemon-fz99s" (UID: "8a0fd497-acc1-4391-bb37-3a00c2c3362d") : secret "metrics-daemon-secret" not found Jul 12 19:20:57.754817 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:57.754777 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/697ae350-ebe5-4859-a7d4-389fd1da28f8-original-pull-secret\") pod \"global-pull-secret-syncer-nd2b6\" (UID: \"697ae350-ebe5-4859-a7d4-389fd1da28f8\") " pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:20:57.821317 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:57.821279 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd2b6" Jul 12 19:20:57.841473 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:57.841448 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25c9d\" (UniqueName: \"kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d\") pod \"network-check-target-5gknd\" (UID: \"a2c5b5b8-f936-4c30-99f8-9cc04474d080\") " pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:20:57.843940 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:57.843921 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 12 19:20:57.854453 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:57.854431 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 12 19:20:57.864930 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:57.864893 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25c9d\" (UniqueName: \"kubernetes.io/projected/a2c5b5b8-f936-4c30-99f8-9cc04474d080-kube-api-access-25c9d\") pod \"network-check-target-5gknd\" (UID: \"a2c5b5b8-f936-4c30-99f8-9cc04474d080\") " pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:20:57.945775 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:57.945742 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nd2b6"] Jul 12 19:20:57.948562 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:20:57.948537 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod697ae350_ebe5_4859_a7d4_389fd1da28f8.slice/crio-d5d593d5ea0cc65ff1352cfc949beb7d882e59f35ad4472e937c1787f8f0b3f3 WatchSource:0}: Error finding container d5d593d5ea0cc65ff1352cfc949beb7d882e59f35ad4472e937c1787f8f0b3f3: Status 404 returned error can't find the container with id d5d593d5ea0cc65ff1352cfc949beb7d882e59f35ad4472e937c1787f8f0b3f3 Jul 12 19:20:58.135302 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:58.135213 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zkgdj\"" Jul 12 19:20:58.143254 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:58.143235 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:20:58.143596 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:58.143571 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-kvdtw\" (UID: \"5efb57d2-5bf0-4799-8e22-3a3066c94cb2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" Jul 12 19:20:58.143664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:58.143647 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:20:58.143704 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:58.143691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:20:58.143738 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:58.143718 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 12 19:20:58.143787 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:58.143776 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert podName:5efb57d2-5bf0-4799-8e22-3a3066c94cb2 nodeName:}" failed. No retries permitted until 2026-07-12 19:21:30.143758621 +0000 UTC m=+98.737032714 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-kvdtw" (UID: "5efb57d2-5bf0-4799-8e22-3a3066c94cb2") : secret "networking-console-plugin-cert" not found Jul 12 19:20:58.143831 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:58.143782 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:20:58.143865 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:58.143829 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls podName:174905d8-e6cd-4e9b-a6fa-02d6f96d05d5 nodeName:}" failed. No retries permitted until 2026-07-12 19:21:30.143819623 +0000 UTC m=+98.737093699 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls") pod "dns-default-zvbnn" (UID: "174905d8-e6cd-4e9b-a6fa-02d6f96d05d5") : secret "dns-default-metrics-tls" not found Jul 12 19:20:58.143865 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:58.143777 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 12 19:20:58.143865 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:58.143849 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7449988cff-n48gz: secret "image-registry-tls" not found Jul 12 19:20:58.143955 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:20:58.143877 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls podName:418ab7c7-cad9-4e69-bde5-d16e643b12d7 nodeName:}" failed. No retries permitted until 2026-07-12 19:21:30.143867919 +0000 UTC m=+98.737141995 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls") pod "image-registry-7449988cff-n48gz" (UID: "418ab7c7-cad9-4e69-bde5-d16e643b12d7") : secret "image-registry-tls" not found Jul 12 19:20:58.224358 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:58.224327 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nd2b6" event={"ID":"697ae350-ebe5-4859-a7d4-389fd1da28f8","Type":"ContainerStarted","Data":"d5d593d5ea0cc65ff1352cfc949beb7d882e59f35ad4472e937c1787f8f0b3f3"} Jul 12 19:20:58.267191 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:58.267076 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5gknd"] Jul 12 19:20:58.269654 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:20:58.269623 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c5b5b8_f936_4c30_99f8_9cc04474d080.slice/crio-868849ae1a376489641a4ed36b89dea86ceb0a3a30ec1cf7c623ead80a4a365e WatchSource:0}: Error finding container 868849ae1a376489641a4ed36b89dea86ceb0a3a30ec1cf7c623ead80a4a365e: Status 404 returned error can't find the container with id 868849ae1a376489641a4ed36b89dea86ceb0a3a30ec1cf7c623ead80a4a365e Jul 12 19:20:59.228721 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:20:59.228682 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5gknd" event={"ID":"a2c5b5b8-f936-4c30-99f8-9cc04474d080","Type":"ContainerStarted","Data":"868849ae1a376489641a4ed36b89dea86ceb0a3a30ec1cf7c623ead80a4a365e"} Jul 12 19:21:02.333202 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:21:02.333160 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:21:02.390923 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:21:02.390878 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:21:02.491792 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:21:02.491757 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:21:02.592795 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:21:02.592709 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:21:03.243104 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:21:03.243000 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nd2b6" event={"ID":"697ae350-ebe5-4859-a7d4-389fd1da28f8","Type":"ContainerStarted","Data":"d3acd1ad51a495c25698b140a55eea2c6e07c9049064820afe3b6b8aab94bfa1"} Jul 12 19:21:03.244607 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:21:03.244576 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5gknd" event={"ID":"a2c5b5b8-f936-4c30-99f8-9cc04474d080","Type":"ContainerStarted","Data":"c450ceb80627cb5a378490da9b0e83d5a5bc86c5240d90fc0e23c32154b8f5bf"} Jul 12 19:21:03.244750 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:21:03.244729 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:21:03.270766 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:21:03.270717 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5gknd" podStartSLOduration=66.574241779 podStartE2EDuration="1m11.270703271s" totalImagesPullingTime="4.696461492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:21:03.270686415 +0000 UTC m=+71.863960519" watchObservedRunningTime="2026-07-12 19:21:03.270703271 +0000 UTC m=+71.863977368" Jul 12 19:21:03.271314 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:21:03.271269 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nd2b6" podStartSLOduration=66.252772287 podStartE2EDuration="1m11.271256224s" totalImagesPullingTime="5.018483937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:21:03.256687046 +0000 UTC m=+71.849961145" watchObservedRunningTime="2026-07-12 19:21:03.271256224 +0000 UTC m=+71.864530322" Jul 12 19:21:06.469438 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:21:06.469404 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:21:06.520289 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:21:06.520262 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:21:29.681428 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:21:29.681313 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:21:30.195824 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:21:30.195790 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-kvdtw\" (UID: \"5efb57d2-5bf0-4799-8e22-3a3066c94cb2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" Jul 12 19:21:30.196065 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:21:30.195877 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:21:30.196065 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:21:30.195914 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:21:30.196065 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:21:30.195947 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 12 19:21:30.196065 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:21:30.196014 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 12 19:21:30.196065 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:21:30.196043 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert podName:5efb57d2-5bf0-4799-8e22-3a3066c94cb2 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:34.196008485 +0000 UTC m=+162.789282563 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-kvdtw" (UID: "5efb57d2-5bf0-4799-8e22-3a3066c94cb2") : secret "networking-console-plugin-cert" not found Jul 12 19:21:30.196065 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:21:30.196050 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7449988cff-n48gz: secret "image-registry-tls" not found Jul 12 19:21:30.196313 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:21:30.196102 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls podName:418ab7c7-cad9-4e69-bde5-d16e643b12d7 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:34.19608871 +0000 UTC m=+162.789362785 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls") pod "image-registry-7449988cff-n48gz" (UID: "418ab7c7-cad9-4e69-bde5-d16e643b12d7") : secret "image-registry-tls" not found Jul 12 19:21:30.196313 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:21:30.196043 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:21:30.196313 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:21:30.196133 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls podName:174905d8-e6cd-4e9b-a6fa-02d6f96d05d5 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:34.196126768 +0000 UTC m=+162.789400844 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls") pod "dns-default-zvbnn" (UID: "174905d8-e6cd-4e9b-a6fa-02d6f96d05d5") : secret "dns-default-metrics-tls" not found Jul 12 19:21:30.600994 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:21:30.600961 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:21:34.251040 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:21:34.250984 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5gknd" Jul 12 19:22:01.838667 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:22:01.838627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs\") pod \"network-metrics-daemon-fz99s\" (UID: \"8a0fd497-acc1-4391-bb37-3a00c2c3362d\") " pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:22:01.839203 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:22:01.838783 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 12 19:22:01.839203 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:22:01.838864 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs podName:8a0fd497-acc1-4391-bb37-3a00c2c3362d nodeName:}" failed. No retries permitted until 2026-07-12 19:24:03.838847263 +0000 UTC m=+252.432121352 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs") pod "network-metrics-daemon-fz99s" (UID: "8a0fd497-acc1-4391-bb37-3a00c2c3362d") : secret "metrics-daemon-secret" not found Jul 12 19:22:29.280162 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:22:29.280105 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7449988cff-n48gz" podUID="418ab7c7-cad9-4e69-bde5-d16e643b12d7" Jul 12 19:22:29.289306 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:22:29.289259 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" podUID="5efb57d2-5bf0-4799-8e22-3a3066c94cb2" Jul 12 19:22:29.349774 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:22:29.349737 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zvbnn" podUID="174905d8-e6cd-4e9b-a6fa-02d6f96d05d5" Jul 12 19:22:29.497801 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:22:29.497764 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:22:29.497801 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:22:29.497794 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zvbnn" Jul 12 19:22:29.498060 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:22:29.497908 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" Jul 12 19:22:29.927776 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:22:29.927736 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-fz99s" podUID="8a0fd497-acc1-4391-bb37-3a00c2c3362d" Jul 12 19:22:34.280304 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:22:34.280252 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:22:34.280304 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:22:34.280307 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:22:34.280808 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:22:34.280344 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-kvdtw\" (UID: \"5efb57d2-5bf0-4799-8e22-3a3066c94cb2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" Jul 12 19:22:34.280808 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:22:34.280400 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 12 19:22:34.280808 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:22:34.280421 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7449988cff-n48gz: secret "image-registry-tls" not found Jul 12 19:22:34.280808 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:22:34.280428 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 12 19:22:34.280808 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:22:34.280453 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:22:34.280808 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:22:34.280487 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert podName:5efb57d2-5bf0-4799-8e22-3a3066c94cb2 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:36.280471496 +0000 UTC m=+284.873745572 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-kvdtw" (UID: "5efb57d2-5bf0-4799-8e22-3a3066c94cb2") : secret "networking-console-plugin-cert" not found Jul 12 19:22:34.280808 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:22:34.280500 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls podName:418ab7c7-cad9-4e69-bde5-d16e643b12d7 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:36.280494388 +0000 UTC m=+284.873768463 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls") pod "image-registry-7449988cff-n48gz" (UID: "418ab7c7-cad9-4e69-bde5-d16e643b12d7") : secret "image-registry-tls" not found Jul 12 19:22:34.280808 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:22:34.280509 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls podName:174905d8-e6cd-4e9b-a6fa-02d6f96d05d5 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:36.28050412 +0000 UTC m=+284.873778196 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls") pod "dns-default-zvbnn" (UID: "174905d8-e6cd-4e9b-a6fa-02d6f96d05d5") : secret "dns-default-metrics-tls" not found Jul 12 19:22:37.153804 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:22:37.153747 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" podUID="ed03d0de-6326-40fa-afc9-639266c9b5f8" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.12:8000/readyz\": dial tcp 10.132.0.12:8000: connect: connection refused" Jul 12 19:22:37.520499 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:22:37.520405 2564 generic.go:356] "Generic (PLEG): container finished" podID="ed03d0de-6326-40fa-afc9-639266c9b5f8" containerID="204b98032bd924a5b28e8520b26e42e1d5ca0dbf40774ca432f3b7555f3a5fde" exitCode=1 Jul 12 19:22:37.520499 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:22:37.520477 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" event={"ID":"ed03d0de-6326-40fa-afc9-639266c9b5f8","Type":"ContainerDied","Data":"204b98032bd924a5b28e8520b26e42e1d5ca0dbf40774ca432f3b7555f3a5fde"} Jul 12 19:22:37.520868 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:22:37.520852 2564 scope.go:117] "RemoveContainer" containerID="204b98032bd924a5b28e8520b26e42e1d5ca0dbf40774ca432f3b7555f3a5fde" Jul 12 19:22:37.522008 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:22:37.521983 2564 generic.go:356] "Generic (PLEG): container finished" podID="2da3d858-73b3-4a08-8678-01155c9115e7" containerID="bdcf6e5c0edfa75e01746737560de840d17d169b5be729bce026c4bc6f1a3fad" exitCode=255 Jul 12 19:22:37.522129 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:22:37.522009 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp" event={"ID":"2da3d858-73b3-4a08-8678-01155c9115e7","Type":"ContainerDied","Data":"bdcf6e5c0edfa75e01746737560de840d17d169b5be729bce026c4bc6f1a3fad"} Jul 12 19:22:37.522333 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:22:37.522317 2564 scope.go:117] "RemoveContainer" containerID="bdcf6e5c0edfa75e01746737560de840d17d169b5be729bce026c4bc6f1a3fad" Jul 12 19:22:38.526106 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:22:38.526064 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp" event={"ID":"2da3d858-73b3-4a08-8678-01155c9115e7","Type":"ContainerStarted","Data":"d1b79f71ee36e2a3ad4734af2159b461780903f73b5125c0e03145425b2cbf9b"} Jul 12 19:22:38.527766 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:22:38.527742 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" event={"ID":"ed03d0de-6326-40fa-afc9-639266c9b5f8","Type":"ContainerStarted","Data":"801c75a64df96fd5bff0c8f3dba9a78fc848f8d02b34bdacfdd669600bb85f56"} Jul 12 19:22:38.527972 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:22:38.527956 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" Jul 12 19:22:38.528580 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:22:38.528563 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" Jul 12 19:22:41.912212 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:22:41.912177 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:23:37.039183 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:23:37.039150 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:03.840794 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:03.840742 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs\") pod \"network-metrics-daemon-fz99s\" (UID: \"8a0fd497-acc1-4391-bb37-3a00c2c3362d\") " pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:24:03.841212 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:03.840906 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 12 19:24:03.841212 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:03.840981 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs podName:8a0fd497-acc1-4391-bb37-3a00c2c3362d nodeName:}" failed. No retries permitted until 2026-07-12 19:26:05.84096373 +0000 UTC m=+374.434237806 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs") pod "network-metrics-daemon-fz99s" (UID: "8a0fd497-acc1-4391-bb37-3a00c2c3362d") : secret "metrics-daemon-secret" not found Jul 12 19:24:07.996608 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:07.996569 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:08.097536 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:08.097504 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:09.818168 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:09.818127 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:09.918344 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:09.918312 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:12.283168 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.283083 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dtmkq"] Jul 12 19:24:12.285313 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.285289 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:12.288856 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.288824 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 12 19:24:12.288856 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.288846 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 12 19:24:12.289890 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.289871 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jgmpf\"" Jul 12 19:24:12.289947 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.289896 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 12 19:24:12.289995 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.289871 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 12 19:24:12.300388 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.300358 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dtmkq"] Jul 12 19:24:12.312524 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.312495 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:12.366212 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.366177 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:12.400385 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.400360 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46d9ad51-4ee6-44b6-9416-5f161034d19f-crio-socket\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:12.400478 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.400410 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhkjh\" (UniqueName: \"kubernetes.io/projected/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-api-access-hhkjh\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:12.400478 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.400430 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:12.400547 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.400518 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46d9ad51-4ee6-44b6-9416-5f161034d19f-data-volume\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:12.400580 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.400544 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:12.501371 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.501335 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhkjh\" (UniqueName: \"kubernetes.io/projected/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-api-access-hhkjh\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:12.501371 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.501374 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:12.501610 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.501439 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46d9ad51-4ee6-44b6-9416-5f161034d19f-data-volume\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:12.501610 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.501463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:12.501610 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:12.501533 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 12 19:24:12.501610 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:12.501541 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 12 19:24:12.501610 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:12.501602 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-rbac-proxy-cm podName:46d9ad51-4ee6-44b6-9416-5f161034d19f nodeName:}" failed. No retries permitted until 2026-07-12 19:24:13.001586009 +0000 UTC m=+261.594860085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtmkq" (UID: "46d9ad51-4ee6-44b6-9416-5f161034d19f") : configmap "kube-rbac-proxy" not found Jul 12 19:24:12.501864 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.501655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46d9ad51-4ee6-44b6-9416-5f161034d19f-crio-socket\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:12.501864 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:12.501666 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls podName:46d9ad51-4ee6-44b6-9416-5f161034d19f nodeName:}" failed. No retries permitted until 2026-07-12 19:24:13.00164999 +0000 UTC m=+261.594924081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtmkq" (UID: "46d9ad51-4ee6-44b6-9416-5f161034d19f") : secret "insights-runtime-extractor-tls" not found Jul 12 19:24:12.501864 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.501698 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46d9ad51-4ee6-44b6-9416-5f161034d19f-crio-socket\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:12.501864 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.501836 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46d9ad51-4ee6-44b6-9416-5f161034d19f-data-volume\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:12.510079 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.510053 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:12.515842 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.515821 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhkjh\" (UniqueName: \"kubernetes.io/projected/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-api-access-hhkjh\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:12.575419 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.575332 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:12.626695 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:12.626657 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:13.006134 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:13.006097 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:13.006307 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:13.006154 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:13.006307 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:13.006216 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 12 19:24:13.006307 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:13.006279 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 12 19:24:13.006307 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:13.006284 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-rbac-proxy-cm podName:46d9ad51-4ee6-44b6-9416-5f161034d19f nodeName:}" failed. No retries permitted until 2026-07-12 19:24:14.00626791 +0000 UTC m=+262.599541989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtmkq" (UID: "46d9ad51-4ee6-44b6-9416-5f161034d19f") : configmap "kube-rbac-proxy" not found Jul 12 19:24:13.006449 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:13.006326 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls podName:46d9ad51-4ee6-44b6-9416-5f161034d19f nodeName:}" failed. No retries permitted until 2026-07-12 19:24:14.006315984 +0000 UTC m=+262.599590060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtmkq" (UID: "46d9ad51-4ee6-44b6-9416-5f161034d19f") : secret "insights-runtime-extractor-tls" not found Jul 12 19:24:13.142034 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:13.141975 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:13.191147 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:13.191112 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:13.574489 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:13.574453 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:13.674147 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:13.674112 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:13.847471 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:13.847382 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:13.948760 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:13.948724 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:14.013298 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:14.013263 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:14.013465 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:14.013346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:14.013465 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:14.013409 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 12 19:24:14.013545 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:14.013465 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 12 19:24:14.013545 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:14.013508 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-rbac-proxy-cm podName:46d9ad51-4ee6-44b6-9416-5f161034d19f nodeName:}" failed. No retries permitted until 2026-07-12 19:24:16.013482267 +0000 UTC m=+264.606756345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtmkq" (UID: "46d9ad51-4ee6-44b6-9416-5f161034d19f") : configmap "kube-rbac-proxy" not found Jul 12 19:24:14.013545 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:14.013531 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls podName:46d9ad51-4ee6-44b6-9416-5f161034d19f nodeName:}" failed. No retries permitted until 2026-07-12 19:24:16.013516759 +0000 UTC m=+264.606790853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtmkq" (UID: "46d9ad51-4ee6-44b6-9416-5f161034d19f") : secret "insights-runtime-extractor-tls" not found Jul 12 19:24:14.072337 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:14.072312 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-grbbl_9a01d1fa-13e1-47af-b3af-2a9de7431dd1/dns-node-resolver/0.log" Jul 12 19:24:14.910161 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:14.910115 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:15.473134 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:15.473107 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-thbcj_ab20a71c-5305-4984-8c11-e3b5e5451fcb/node-ca/0.log" Jul 12 19:24:15.479623 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:15.479600 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:15.580319 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:15.580290 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:16.027329 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:16.027283 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:16.027711 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:16.027352 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:16.027711 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:16.027414 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 12 19:24:16.027711 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:16.027463 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 12 19:24:16.027711 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:16.027484 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-rbac-proxy-cm podName:46d9ad51-4ee6-44b6-9416-5f161034d19f nodeName:}" failed. No retries permitted until 2026-07-12 19:24:20.027467368 +0000 UTC m=+268.620741467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtmkq" (UID: "46d9ad51-4ee6-44b6-9416-5f161034d19f") : configmap "kube-rbac-proxy" not found Jul 12 19:24:16.027711 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:16.027502 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls podName:46d9ad51-4ee6-44b6-9416-5f161034d19f nodeName:}" failed. No retries permitted until 2026-07-12 19:24:20.027491514 +0000 UTC m=+268.620765589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtmkq" (UID: "46d9ad51-4ee6-44b6-9416-5f161034d19f") : secret "insights-runtime-extractor-tls" not found Jul 12 19:24:16.073647 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:16.073624 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9c99k_b6b3ac65-ccd5-4d7b-8809-8372731b54be/serve-healthcheck-canary/0.log" Jul 12 19:24:16.370185 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:16.370108 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:18.495986 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:18.495949 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:18.783444 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:18.783366 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:18.920744 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:18.920713 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:18.968270 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:18.968236 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:19.057582 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:19.057500 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:19.300066 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:19.300014 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:19.490855 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:19.490816 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:20.055386 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:20.055351 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:20.055798 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:20.055426 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:20.055798 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:20.055487 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 12 19:24:20.055798 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:20.055496 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 12 19:24:20.055798 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:20.055545 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls podName:46d9ad51-4ee6-44b6-9416-5f161034d19f nodeName:}" failed. No retries permitted until 2026-07-12 19:24:28.055530701 +0000 UTC m=+276.648804778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtmkq" (UID: "46d9ad51-4ee6-44b6-9416-5f161034d19f") : secret "insights-runtime-extractor-tls" not found Jul 12 19:24:20.055798 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:20.055558 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-rbac-proxy-cm podName:46d9ad51-4ee6-44b6-9416-5f161034d19f nodeName:}" failed. No retries permitted until 2026-07-12 19:24:28.055552178 +0000 UTC m=+276.648826254 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtmkq" (UID: "46d9ad51-4ee6-44b6-9416-5f161034d19f") : configmap "kube-rbac-proxy" not found Jul 12 19:24:20.487615 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:20.487574 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:20.687801 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:20.687764 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:20.886377 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:20.886291 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:21.285941 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:21.285905 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:21.886244 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:21.886210 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:22.485943 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:22.485899 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:22.537161 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:22.537124 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:22.685449 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:22.685416 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:22.736607 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:22.736529 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:22.820524 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:22.820491 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:22.921270 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:22.921233 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:23.289742 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:23.289711 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:24.089714 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:24.089677 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:24.295589 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:24.295551 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:24.687883 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:24.687838 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:24.886568 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:24.886538 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:25.202585 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:25.202549 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:25.302998 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:25.302961 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:25.523501 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:25.523417 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:25.977271 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:25.977229 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:26.517654 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:26.517620 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:26.683614 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:26.683577 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:26.882637 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:26.882545 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:27.286217 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:27.286179 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:27.688347 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:27.688307 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:28.118377 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:28.118285 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:28.118377 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:28.118345 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:28.118611 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:28.118523 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 12 19:24:28.118611 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:28.118604 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls podName:46d9ad51-4ee6-44b6-9416-5f161034d19f nodeName:}" failed. No retries permitted until 2026-07-12 19:24:44.118581074 +0000 UTC m=+292.711855170 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtmkq" (UID: "46d9ad51-4ee6-44b6-9416-5f161034d19f") : secret "insights-runtime-extractor-tls" not found Jul 12 19:24:28.118851 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:28.118831 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46d9ad51-4ee6-44b6-9416-5f161034d19f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:28.282919 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:28.282875 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:28.891703 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:28.891662 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:28.948973 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:28.948936 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:29.094674 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:29.094641 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:29.286696 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:29.286664 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:29.483675 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:29.483637 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:32.498918 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:32.498869 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zvbnn" podUID="174905d8-e6cd-4e9b-a6fa-02d6f96d05d5" Jul 12 19:24:32.498918 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:32.498870 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" podUID="5efb57d2-5bf0-4799-8e22-3a3066c94cb2" Jul 12 19:24:32.498918 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:32.498869 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7449988cff-n48gz" podUID="418ab7c7-cad9-4e69-bde5-d16e643b12d7" Jul 12 19:24:32.858458 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:32.858377 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:24:32.858585 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:32.858382 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zvbnn" Jul 12 19:24:32.858585 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:32.858382 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" Jul 12 19:24:34.911892 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:34.911860 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:35.018556 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:35.018517 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:36.283852 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.283800 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:24:36.283852 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.283864 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:24:36.284414 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.283912 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-kvdtw\" (UID: \"5efb57d2-5bf0-4799-8e22-3a3066c94cb2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" Jul 12 19:24:36.286320 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.286296 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/174905d8-e6cd-4e9b-a6fa-02d6f96d05d5-metrics-tls\") pod \"dns-default-zvbnn\" (UID: \"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5\") " pod="openshift-dns/dns-default-zvbnn" Jul 12 19:24:36.286462 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.286410 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls\") pod \"image-registry-7449988cff-n48gz\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:24:36.286462 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.286415 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5efb57d2-5bf0-4799-8e22-3a3066c94cb2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-kvdtw\" (UID: \"5efb57d2-5bf0-4799-8e22-3a3066c94cb2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" Jul 12 19:24:36.462099 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.462070 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-29mst\"" Jul 12 19:24:36.462099 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.462095 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qvb27\"" Jul 12 19:24:36.462295 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.462123 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-sdv2p\"" Jul 12 19:24:36.469267 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.469245 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zvbnn" Jul 12 19:24:36.469267 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.469262 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:24:36.469361 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.469263 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" Jul 12 19:24:36.620244 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.620214 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zvbnn"] Jul 12 19:24:36.624365 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:24:36.624328 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174905d8_e6cd_4e9b_a6fa_02d6f96d05d5.slice/crio-2b3b45ec52cb2889084298782f696dacac72477a1a273955cd1f49117bbf2d58 WatchSource:0}: Error finding container 2b3b45ec52cb2889084298782f696dacac72477a1a273955cd1f49117bbf2d58: Status 404 returned error can't find the container with id 2b3b45ec52cb2889084298782f696dacac72477a1a273955cd1f49117bbf2d58 Jul 12 19:24:36.639487 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.639444 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7449988cff-n48gz"] Jul 12 19:24:36.641981 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:24:36.641961 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418ab7c7_cad9_4e69_bde5_d16e643b12d7.slice/crio-798977655b8f659933b1324707b22128d90573df976db13429721f47cacb039d WatchSource:0}: Error finding container 798977655b8f659933b1324707b22128d90573df976db13429721f47cacb039d: Status 404 returned error can't find the container with id 798977655b8f659933b1324707b22128d90573df976db13429721f47cacb039d Jul 12 19:24:36.651922 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.651898 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-kvdtw"] Jul 12 19:24:36.654054 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:24:36.654010 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5efb57d2_5bf0_4799_8e22_3a3066c94cb2.slice/crio-95847c314e4c7cf9094176af63199a80c2053b790e2fa9ec8b3bc0d1f34ab0b0 WatchSource:0}: Error finding container 95847c314e4c7cf9094176af63199a80c2053b790e2fa9ec8b3bc0d1f34ab0b0: Status 404 returned error can't find the container with id 95847c314e4c7cf9094176af63199a80c2053b790e2fa9ec8b3bc0d1f34ab0b0 Jul 12 19:24:36.871458 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.871372 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zvbnn" event={"ID":"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5","Type":"ContainerStarted","Data":"2b3b45ec52cb2889084298782f696dacac72477a1a273955cd1f49117bbf2d58"} Jul 12 19:24:36.872550 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.872522 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" event={"ID":"5efb57d2-5bf0-4799-8e22-3a3066c94cb2","Type":"ContainerStarted","Data":"95847c314e4c7cf9094176af63199a80c2053b790e2fa9ec8b3bc0d1f34ab0b0"} Jul 12 19:24:36.873968 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.873939 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7449988cff-n48gz" event={"ID":"418ab7c7-cad9-4e69-bde5-d16e643b12d7","Type":"ContainerStarted","Data":"28d0b88583821a16af031d8f5e7bf91e397c7f2fe1b1262ced6c4a7be9efa3ee"} Jul 12 19:24:36.874084 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.873989 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7449988cff-n48gz" event={"ID":"418ab7c7-cad9-4e69-bde5-d16e643b12d7","Type":"ContainerStarted","Data":"798977655b8f659933b1324707b22128d90573df976db13429721f47cacb039d"} Jul 12 19:24:36.874084 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.874054 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:24:36.892073 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:36.892006 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7449988cff-n48gz" podStartSLOduration=284.891991358 podStartE2EDuration="4m44.891991358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:24:36.890796849 +0000 UTC m=+285.484070948" watchObservedRunningTime="2026-07-12 19:24:36.891991358 +0000 UTC m=+285.485265455" Jul 12 19:24:37.879490 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:37.879414 2564 generic.go:356] "Generic (PLEG): container finished" podID="ed03d0de-6326-40fa-afc9-639266c9b5f8" containerID="801c75a64df96fd5bff0c8f3dba9a78fc848f8d02b34bdacfdd669600bb85f56" exitCode=1 Jul 12 19:24:37.879914 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:37.879489 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" event={"ID":"ed03d0de-6326-40fa-afc9-639266c9b5f8","Type":"ContainerDied","Data":"801c75a64df96fd5bff0c8f3dba9a78fc848f8d02b34bdacfdd669600bb85f56"} Jul 12 19:24:37.879914 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:37.879557 2564 scope.go:117] "RemoveContainer" containerID="204b98032bd924a5b28e8520b26e42e1d5ca0dbf40774ca432f3b7555f3a5fde" Jul 12 19:24:37.879914 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:37.879887 2564 scope.go:117] "RemoveContainer" containerID="801c75a64df96fd5bff0c8f3dba9a78fc848f8d02b34bdacfdd669600bb85f56" Jul 12 19:24:37.880169 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:37.880145 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-86c47f4989-v7tnh_open-cluster-management-agent-addon(ed03d0de-6326-40fa-afc9-639266c9b5f8)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" podUID="ed03d0de-6326-40fa-afc9-639266c9b5f8" Jul 12 19:24:37.881400 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:37.881379 2564 generic.go:356] "Generic (PLEG): container finished" podID="2da3d858-73b3-4a08-8678-01155c9115e7" containerID="d1b79f71ee36e2a3ad4734af2159b461780903f73b5125c0e03145425b2cbf9b" exitCode=255 Jul 12 19:24:37.881495 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:37.881404 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp" event={"ID":"2da3d858-73b3-4a08-8678-01155c9115e7","Type":"ContainerDied","Data":"d1b79f71ee36e2a3ad4734af2159b461780903f73b5125c0e03145425b2cbf9b"} Jul 12 19:24:37.881729 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:37.881715 2564 scope.go:117] "RemoveContainer" containerID="d1b79f71ee36e2a3ad4734af2159b461780903f73b5125c0e03145425b2cbf9b" Jul 12 19:24:37.881916 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:37.881873 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-7496b85cbd-t5xcp_open-cluster-management-agent-addon(2da3d858-73b3-4a08-8678-01155c9115e7)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp" podUID="2da3d858-73b3-4a08-8678-01155c9115e7" Jul 12 19:24:37.883070 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:37.883010 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" event={"ID":"5efb57d2-5bf0-4799-8e22-3a3066c94cb2","Type":"ContainerStarted","Data":"79e8d28b58068bee311d2e84c63e5eb0bd9cb502f76ec30a20cdfa605d1978eb"} Jul 12 19:24:37.926580 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:37.926529 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-kvdtw" podStartSLOduration=277.796539032 podStartE2EDuration="4m38.926513642s" totalImagesPullingTime="1.12997461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:24:37.925629314 +0000 UTC m=+286.518903413" watchObservedRunningTime="2026-07-12 19:24:37.926513642 +0000 UTC m=+286.519787741" Jul 12 19:24:38.009008 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:38.008971 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:38.066289 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:38.066243 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:38.299397 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:38.299374 2564 scope.go:117] "RemoveContainer" containerID="bdcf6e5c0edfa75e01746737560de840d17d169b5be729bce026c4bc6f1a3fad" Jul 12 19:24:38.528185 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:38.528151 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" Jul 12 19:24:38.888747 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:38.888705 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zvbnn" event={"ID":"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5","Type":"ContainerStarted","Data":"6cc55e9e96d98b42c1b3f984739443dc2ad8358794391a9ed920c92a3680d0ca"} Jul 12 19:24:38.889286 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:38.888757 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zvbnn" event={"ID":"174905d8-e6cd-4e9b-a6fa-02d6f96d05d5","Type":"ContainerStarted","Data":"bf476c2517765f5104d168998dededcc1aee84a330aaa526a6f70a97028b0e43"} Jul 12 19:24:38.889286 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:38.888824 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zvbnn" Jul 12 19:24:38.890402 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:38.890386 2564 scope.go:117] "RemoveContainer" containerID="801c75a64df96fd5bff0c8f3dba9a78fc848f8d02b34bdacfdd669600bb85f56" Jul 12 19:24:38.890533 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:38.890515 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-86c47f4989-v7tnh_open-cluster-management-agent-addon(ed03d0de-6326-40fa-afc9-639266c9b5f8)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" podUID="ed03d0de-6326-40fa-afc9-639266c9b5f8" Jul 12 19:24:38.905472 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:38.905414 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zvbnn" podStartSLOduration=251.206543168 podStartE2EDuration="4m12.905401404s" totalImagesPullingTime="1.698858236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:20:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:24:38.904341247 +0000 UTC m=+287.497615344" watchObservedRunningTime="2026-07-12 19:24:38.905401404 +0000 UTC m=+287.498675479" Jul 12 19:24:41.915422 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:41.915382 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:42.014687 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:42.014653 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:42.074000 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:42.073960 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:42.222053 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:42.221947 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:44.092042 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:44.091984 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:44.149831 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:44.149795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:44.152238 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:44.152206 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46d9ad51-4ee6-44b6-9416-5f161034d19f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtmkq\" (UID: \"46d9ad51-4ee6-44b6-9416-5f161034d19f\") " pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:44.153506 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:44.153483 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:44.365059 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:44.364948 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:44.393997 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:44.393962 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dtmkq" Jul 12 19:24:44.521038 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:44.520918 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dtmkq"] Jul 12 19:24:44.523621 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:24:44.523594 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d9ad51_4ee6_44b6_9416_5f161034d19f.slice/crio-a351451a0ff2ae4be0a5d3e494e8d636061747ec74f99f70cbce473cdce0d91d WatchSource:0}: Error finding container a351451a0ff2ae4be0a5d3e494e8d636061747ec74f99f70cbce473cdce0d91d: Status 404 returned error can't find the container with id a351451a0ff2ae4be0a5d3e494e8d636061747ec74f99f70cbce473cdce0d91d Jul 12 19:24:44.911036 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:44.910968 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dtmkq" event={"ID":"46d9ad51-4ee6-44b6-9416-5f161034d19f","Type":"ContainerStarted","Data":"a351451a0ff2ae4be0a5d3e494e8d636061747ec74f99f70cbce473cdce0d91d"} Jul 12 19:24:44.913376 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:44.913337 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-fz99s" podUID="8a0fd497-acc1-4391-bb37-3a00c2c3362d" Jul 12 19:24:46.303956 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:46.303915 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:46.611229 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:46.611146 2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp" Jul 12 19:24:46.611586 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:46.611562 2564 scope.go:117] "RemoveContainer" containerID="d1b79f71ee36e2a3ad4734af2159b461780903f73b5125c0e03145425b2cbf9b" Jul 12 19:24:46.611762 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:46.611739 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-7496b85cbd-t5xcp_open-cluster-management-agent-addon(2da3d858-73b3-4a08-8678-01155c9115e7)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp" podUID="2da3d858-73b3-4a08-8678-01155c9115e7" Jul 12 19:24:46.618325 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:46.618298 2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" Jul 12 19:24:46.618704 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:46.618684 2564 scope.go:117] "RemoveContainer" containerID="801c75a64df96fd5bff0c8f3dba9a78fc848f8d02b34bdacfdd669600bb85f56" Jul 12 19:24:46.618884 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:46.618858 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-86c47f4989-v7tnh_open-cluster-management-agent-addon(ed03d0de-6326-40fa-afc9-639266c9b5f8)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" podUID="ed03d0de-6326-40fa-afc9-639266c9b5f8" Jul 12 19:24:48.894886 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:48.894858 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zvbnn" Jul 12 19:24:48.927898 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:48.927869 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dtmkq" event={"ID":"46d9ad51-4ee6-44b6-9416-5f161034d19f","Type":"ContainerStarted","Data":"f248c54b00bb8abb56547d3b40ed6ebe20f5c413e5a6924b8625591599fc34d1"} Jul 12 19:24:48.928055 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:48.927925 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dtmkq" event={"ID":"46d9ad51-4ee6-44b6-9416-5f161034d19f","Type":"ContainerStarted","Data":"1fcdead981b7bad54940817608447298b8d6aa1882abca9acf2b5dbeae0486bb"} Jul 12 19:24:49.234360 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.234277 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:49.454821 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.454780 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:49.476341 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.476309 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c8q7b"] Jul 12 19:24:49.478875 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.478852 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.482691 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.482564 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 12 19:24:49.482691 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.482622 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 12 19:24:49.483033 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.482982 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 12 19:24:49.483033 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.483003 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 12 19:24:49.483703 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.483660 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 12 19:24:49.484069 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.484051 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5d6qj\"" Jul 12 19:24:49.484420 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.484371 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 12 19:24:49.549361 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.549319 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:49.591188 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.591153 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0da2150f-4939-40fa-b34e-80957d3b4416-sys\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.591344 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.591193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r2h8\" (UniqueName: \"kubernetes.io/projected/0da2150f-4939-40fa-b34e-80957d3b4416-kube-api-access-6r2h8\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.591344 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.591214 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0da2150f-4939-40fa-b34e-80957d3b4416-node-exporter-tls\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.591344 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.591328 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0da2150f-4939-40fa-b34e-80957d3b4416-metrics-client-ca\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.591479 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.591364 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0da2150f-4939-40fa-b34e-80957d3b4416-node-exporter-accelerators-collector-config\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.591479 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.591433 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0da2150f-4939-40fa-b34e-80957d3b4416-node-exporter-textfile\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.591479 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.591460 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0da2150f-4939-40fa-b34e-80957d3b4416-node-exporter-wtmp\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.591632 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.591502 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0da2150f-4939-40fa-b34e-80957d3b4416-root\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.591632 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.591562 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0da2150f-4939-40fa-b34e-80957d3b4416-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.651348 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.651311 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:49.692346 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.692315 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0da2150f-4939-40fa-b34e-80957d3b4416-sys\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.692527 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.692356 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r2h8\" (UniqueName: \"kubernetes.io/projected/0da2150f-4939-40fa-b34e-80957d3b4416-kube-api-access-6r2h8\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.692527 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.692382 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0da2150f-4939-40fa-b34e-80957d3b4416-node-exporter-tls\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.692527 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.692445 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0da2150f-4939-40fa-b34e-80957d3b4416-sys\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.692527 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.692446 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0da2150f-4939-40fa-b34e-80957d3b4416-metrics-client-ca\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.692748 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:49.692573 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 12 19:24:49.692748 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.692633 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0da2150f-4939-40fa-b34e-80957d3b4416-node-exporter-accelerators-collector-config\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.692748 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:24:49.692648 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0da2150f-4939-40fa-b34e-80957d3b4416-node-exporter-tls podName:0da2150f-4939-40fa-b34e-80957d3b4416 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:50.192628377 +0000 UTC m=+298.785902468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0da2150f-4939-40fa-b34e-80957d3b4416-node-exporter-tls") pod "node-exporter-c8q7b" (UID: "0da2150f-4939-40fa-b34e-80957d3b4416") : secret "node-exporter-tls" not found Jul 12 19:24:49.692748 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.692695 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0da2150f-4939-40fa-b34e-80957d3b4416-node-exporter-textfile\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.692949 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.692756 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0da2150f-4939-40fa-b34e-80957d3b4416-node-exporter-wtmp\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.692949 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.692793 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0da2150f-4939-40fa-b34e-80957d3b4416-root\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.692949 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.692835 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0da2150f-4939-40fa-b34e-80957d3b4416-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.692949 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.692901 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0da2150f-4939-40fa-b34e-80957d3b4416-root\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.692949 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.692906 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0da2150f-4939-40fa-b34e-80957d3b4416-node-exporter-wtmp\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.693227 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.692955 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0da2150f-4939-40fa-b34e-80957d3b4416-node-exporter-textfile\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.693227 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.693153 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0da2150f-4939-40fa-b34e-80957d3b4416-metrics-client-ca\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.693316 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.693224 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0da2150f-4939-40fa-b34e-80957d3b4416-node-exporter-accelerators-collector-config\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.695414 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.695390 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0da2150f-4939-40fa-b34e-80957d3b4416-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.703852 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.703802 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r2h8\" (UniqueName: \"kubernetes.io/projected/0da2150f-4939-40fa-b34e-80957d3b4416-kube-api-access-6r2h8\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:49.753955 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.753779 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:49.857767 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.857723 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:49.976125 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:49.976074 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:50.076672 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:50.076569 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:50.196831 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:50.196789 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0da2150f-4939-40fa-b34e-80957d3b4416-node-exporter-tls\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:50.199433 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:50.199409 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0da2150f-4939-40fa-b34e-80957d3b4416-node-exporter-tls\") pod \"node-exporter-c8q7b\" (UID: \"0da2150f-4939-40fa-b34e-80957d3b4416\") " pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:50.390827 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:50.390743 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c8q7b" Jul 12 19:24:50.435358 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:50.435299 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:50.535388 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:50.535346 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:50.625475 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:50.625443 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:50.759329 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:50.759291 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:50.937742 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:50.937704 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:51.039152 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:51.039009 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:51.138896 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:51.138867 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:51.164331 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:24:51.164304 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da2150f_4939_40fa_b34e_80957d3b4416.slice/crio-29fdcb7904079528ab50997f3f098ea18e6e2553c2678a7c5f4502f607d4858c WatchSource:0}: Error finding container 29fdcb7904079528ab50997f3f098ea18e6e2553c2678a7c5f4502f607d4858c: Status 404 returned error can't find the container with id 29fdcb7904079528ab50997f3f098ea18e6e2553c2678a7c5f4502f607d4858c Jul 12 19:24:51.821539 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:51.821513 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 12 19:24:51.939751 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:51.939584 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c8q7b" event={"ID":"0da2150f-4939-40fa-b34e-80957d3b4416","Type":"ContainerStarted","Data":"29fdcb7904079528ab50997f3f098ea18e6e2553c2678a7c5f4502f607d4858c"} Jul 12 19:24:51.954790 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:51.941688 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dtmkq" event={"ID":"46d9ad51-4ee6-44b6-9416-5f161034d19f","Type":"ContainerStarted","Data":"9e8d2471a0fb4297bbb2ee469efff0a2e5558200abe56533122a92801005fb08"} Jul 12 19:24:51.958345 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:51.958300 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dtmkq" podStartSLOduration=33.38582021 podStartE2EDuration="39.958286672s" totalImagesPullingTime="6.572466462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:24:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:24:51.956775059 +0000 UTC m=+300.550049181" watchObservedRunningTime="2026-07-12 19:24:51.958286672 +0000 UTC m=+300.551560774" Jul 12 19:24:52.386715 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:52.386678 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:52.461271 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:52.461229 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:52.514944 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:52.514890 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:52.920500 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:52.920469 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:52.946657 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:52.946626 2564 generic.go:356] "Generic (PLEG): container finished" podID="0da2150f-4939-40fa-b34e-80957d3b4416" containerID="cbcc864bc1bf7b01cb50cd486b9ca85d81e8678da539d448911a75123fdaeecb" exitCode=0 Jul 12 19:24:52.946815 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:52.946719 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c8q7b" event={"ID":"0da2150f-4939-40fa-b34e-80957d3b4416","Type":"ContainerDied","Data":"cbcc864bc1bf7b01cb50cd486b9ca85d81e8678da539d448911a75123fdaeecb"} Jul 12 19:24:53.951305 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:53.951263 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c8q7b" event={"ID":"0da2150f-4939-40fa-b34e-80957d3b4416","Type":"ContainerStarted","Data":"14cd439f57bbd81a98b1013e1b35bc1e01035c421129099037977ba67745a3d6"} Jul 12 19:24:53.951698 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:53.951316 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c8q7b" event={"ID":"0da2150f-4939-40fa-b34e-80957d3b4416","Type":"ContainerStarted","Data":"de137d81a805cc8772639202fccc8226bce8d8c309086d48601473d9dc59b091"} Jul 12 19:24:53.973935 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:53.973885 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c8q7b" podStartSLOduration=3.921791818 podStartE2EDuration="4.973870231s" totalImagesPullingTime="980.612513ms" totalInitContainerRuntime="71.4659ms" isStatefulPod=true podCreationTimestamp="2026-07-12 19:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:24:53.972136747 +0000 UTC m=+302.565410846" watchObservedRunningTime="2026-07-12 19:24:53.973870231 +0000 UTC m=+302.567144328" Jul 12 19:24:54.252583 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:54.252503 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:55.314529 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:55.314492 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:55.623797 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:55.623717 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:56.473968 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:56.473937 2564 patch_prober.go:28] interesting pod/image-registry-7449988cff-n48gz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 12 19:24:56.474363 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:56.473997 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7449988cff-n48gz" podUID="418ab7c7-cad9-4e69-bde5-d16e643b12d7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 12 19:24:56.910708 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:56.910686 2564 scope.go:117] "RemoveContainer" containerID="801c75a64df96fd5bff0c8f3dba9a78fc848f8d02b34bdacfdd669600bb85f56" Jul 12 19:24:56.953654 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:56.953630 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7449988cff-n48gz"] Jul 12 19:24:56.958048 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:56.957990 2564 patch_prober.go:28] interesting pod/image-registry-7449988cff-n48gz container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 12 19:24:56.958157 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:56.958056 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7449988cff-n48gz" podUID="418ab7c7-cad9-4e69-bde5-d16e643b12d7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 12 19:24:57.028773 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:57.028742 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:57.178427 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:57.178337 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:57.230691 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:57.230662 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:57.968070 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:57.968011 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" event={"ID":"ed03d0de-6326-40fa-afc9-639266c9b5f8","Type":"ContainerStarted","Data":"3746dd051968bcd82cd967015802b4b88f5e081ee7d66fc6be766302cf09ca8f"} Jul 12 19:24:57.968458 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:57.968332 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" Jul 12 19:24:57.969967 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:57.969947 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c47f4989-v7tnh" Jul 12 19:24:58.015893 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:58.015866 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:58.065707 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:58.065667 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:58.910361 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:24:58.910325 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:25:00.911327 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:00.911296 2564 scope.go:117] "RemoveContainer" containerID="d1b79f71ee36e2a3ad4734af2159b461780903f73b5125c0e03145425b2cbf9b" Jul 12 19:25:00.979525 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:00.979492 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7496b85cbd-t5xcp" event={"ID":"2da3d858-73b3-4a08-8678-01155c9115e7","Type":"ContainerStarted","Data":"2706db040e51f8580dd890334f58e8a5de43efbae84b55f45859386e59f0a6ce"} Jul 12 19:25:04.685834 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:04.685799 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:04.786372 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:04.786337 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:04.836588 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:04.836559 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:04.948174 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:04.948088 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:05.007367 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:05.007319 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:05.181412 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:05.181376 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:05.778674 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:05.778636 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:05.832655 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:05.832626 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:06.159253 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:06.159208 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:06.367783 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:06.367753 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:06.560586 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:06.560505 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:06.958345 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:06.958322 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:25:06.978739 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:06.978680 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:07.106284 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:07.106251 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:09.065307 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:09.065269 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:09.115240 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:09.115200 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:10.093459 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:10.093430 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:10.194723 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:10.194682 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:13.924296 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:13.924255 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:14.026136 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:14.026097 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:16.786084 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:16.786047 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:16.886291 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:16.886252 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:18.133722 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:18.133685 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:18.181722 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:18.181686 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:20.581845 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:20.581798 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:20.631115 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:20.631081 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:21.974397 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:21.974362 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7449988cff-n48gz" podUID="418ab7c7-cad9-4e69-bde5-d16e643b12d7" containerName="registry" containerID="cri-o://28d0b88583821a16af031d8f5e7bf91e397c7f2fe1b1262ced6c4a7be9efa3ee" gracePeriod=30 Jul 12 19:25:22.216852 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.216827 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:25:22.331555 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.331513 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-certificates\") pod \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " Jul 12 19:25:22.331555 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.331557 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls\") pod \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " Jul 12 19:25:22.331809 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.331583 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-bound-sa-token\") pod \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " Jul 12 19:25:22.331809 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.331624 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkn2w\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-kube-api-access-wkn2w\") pod \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " Jul 12 19:25:22.331809 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.331647 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/418ab7c7-cad9-4e69-bde5-d16e643b12d7-image-registry-private-configuration\") pod \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " Jul 12 19:25:22.331809 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.331666 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/418ab7c7-cad9-4e69-bde5-d16e643b12d7-ca-trust-extracted\") pod \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " Jul 12 19:25:22.331809 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.331683 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/418ab7c7-cad9-4e69-bde5-d16e643b12d7-trusted-ca\") pod \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " Jul 12 19:25:22.331809 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.331699 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/418ab7c7-cad9-4e69-bde5-d16e643b12d7-installation-pull-secrets\") pod \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\" (UID: \"418ab7c7-cad9-4e69-bde5-d16e643b12d7\") " Jul 12 19:25:22.332138 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.332077 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "418ab7c7-cad9-4e69-bde5-d16e643b12d7" (UID: "418ab7c7-cad9-4e69-bde5-d16e643b12d7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:25:22.332290 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.332238 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/418ab7c7-cad9-4e69-bde5-d16e643b12d7-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "418ab7c7-cad9-4e69-bde5-d16e643b12d7" (UID: "418ab7c7-cad9-4e69-bde5-d16e643b12d7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:25:22.334589 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.334561 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "418ab7c7-cad9-4e69-bde5-d16e643b12d7" (UID: "418ab7c7-cad9-4e69-bde5-d16e643b12d7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:25:22.334809 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.334782 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/418ab7c7-cad9-4e69-bde5-d16e643b12d7-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "418ab7c7-cad9-4e69-bde5-d16e643b12d7" (UID: "418ab7c7-cad9-4e69-bde5-d16e643b12d7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:25:22.334946 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.334874 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-kube-api-access-wkn2w" (OuterVolumeSpecName: "kube-api-access-wkn2w") pod "418ab7c7-cad9-4e69-bde5-d16e643b12d7" (UID: "418ab7c7-cad9-4e69-bde5-d16e643b12d7"). InnerVolumeSpecName "kube-api-access-wkn2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:25:22.334946 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.334900 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/418ab7c7-cad9-4e69-bde5-d16e643b12d7-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "418ab7c7-cad9-4e69-bde5-d16e643b12d7" (UID: "418ab7c7-cad9-4e69-bde5-d16e643b12d7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:25:22.335083 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.335058 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "418ab7c7-cad9-4e69-bde5-d16e643b12d7" (UID: "418ab7c7-cad9-4e69-bde5-d16e643b12d7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:25:22.343597 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.343537 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418ab7c7-cad9-4e69-bde5-d16e643b12d7-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "418ab7c7-cad9-4e69-bde5-d16e643b12d7" (UID: "418ab7c7-cad9-4e69-bde5-d16e643b12d7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:25:22.432167 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.432136 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/418ab7c7-cad9-4e69-bde5-d16e643b12d7-trusted-ca\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:25:22.432167 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.432165 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/418ab7c7-cad9-4e69-bde5-d16e643b12d7-installation-pull-secrets\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:25:22.432364 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.432180 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-certificates\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:25:22.432364 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.432192 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-registry-tls\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:25:22.432364 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.432203 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-bound-sa-token\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:25:22.432364 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.432213 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkn2w\" (UniqueName: \"kubernetes.io/projected/418ab7c7-cad9-4e69-bde5-d16e643b12d7-kube-api-access-wkn2w\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:25:22.432364 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.432224 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/418ab7c7-cad9-4e69-bde5-d16e643b12d7-image-registry-private-configuration\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:25:22.432364 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:22.432236 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/418ab7c7-cad9-4e69-bde5-d16e643b12d7-ca-trust-extracted\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:25:23.046157 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:23.046120 2564 generic.go:356] "Generic (PLEG): container finished" podID="418ab7c7-cad9-4e69-bde5-d16e643b12d7" containerID="28d0b88583821a16af031d8f5e7bf91e397c7f2fe1b1262ced6c4a7be9efa3ee" exitCode=0 Jul 12 19:25:23.046157 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:23.046158 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7449988cff-n48gz" event={"ID":"418ab7c7-cad9-4e69-bde5-d16e643b12d7","Type":"ContainerDied","Data":"28d0b88583821a16af031d8f5e7bf91e397c7f2fe1b1262ced6c4a7be9efa3ee"} Jul 12 19:25:23.046777 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:23.046169 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7449988cff-n48gz" Jul 12 19:25:23.046777 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:23.046197 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7449988cff-n48gz" event={"ID":"418ab7c7-cad9-4e69-bde5-d16e643b12d7","Type":"ContainerDied","Data":"798977655b8f659933b1324707b22128d90573df976db13429721f47cacb039d"} Jul 12 19:25:23.046777 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:23.046222 2564 scope.go:117] "RemoveContainer" containerID="28d0b88583821a16af031d8f5e7bf91e397c7f2fe1b1262ced6c4a7be9efa3ee" Jul 12 19:25:23.054467 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:23.054452 2564 scope.go:117] "RemoveContainer" containerID="28d0b88583821a16af031d8f5e7bf91e397c7f2fe1b1262ced6c4a7be9efa3ee" Jul 12 19:25:23.054711 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:25:23.054693 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d0b88583821a16af031d8f5e7bf91e397c7f2fe1b1262ced6c4a7be9efa3ee\": container with ID starting with 28d0b88583821a16af031d8f5e7bf91e397c7f2fe1b1262ced6c4a7be9efa3ee not found: ID does not exist" containerID="28d0b88583821a16af031d8f5e7bf91e397c7f2fe1b1262ced6c4a7be9efa3ee" Jul 12 19:25:23.054762 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:23.054722 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d0b88583821a16af031d8f5e7bf91e397c7f2fe1b1262ced6c4a7be9efa3ee"} err="failed to get container status \"28d0b88583821a16af031d8f5e7bf91e397c7f2fe1b1262ced6c4a7be9efa3ee\": rpc error: code = NotFound desc = could not find container \"28d0b88583821a16af031d8f5e7bf91e397c7f2fe1b1262ced6c4a7be9efa3ee\": container with ID starting with 28d0b88583821a16af031d8f5e7bf91e397c7f2fe1b1262ced6c4a7be9efa3ee not found: ID does not exist" Jul 12 19:25:23.065511 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:23.065492 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7449988cff-n48gz"] Jul 12 19:25:23.070650 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:23.070630 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7449988cff-n48gz"] Jul 12 19:25:23.914854 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:23.914823 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="418ab7c7-cad9-4e69-bde5-d16e643b12d7" path="/var/lib/kubelet/pods/418ab7c7-cad9-4e69-bde5-d16e643b12d7/volumes" Jul 12 19:25:24.932454 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:24.932428 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c8q7b_0da2150f-4939-40fa-b34e-80957d3b4416/init-textfile/0.log" Jul 12 19:25:24.985803 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:24.985761 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:25.032247 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:25.032210 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:25.129583 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:25.129554 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c8q7b_0da2150f-4939-40fa-b34e-80957d3b4416/node-exporter/0.log" Jul 12 19:25:25.329261 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:25.329195 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c8q7b_0da2150f-4939-40fa-b34e-80957d3b4416/kube-rbac-proxy/0.log" Jul 12 19:25:28.915560 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:28.915527 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:28.967087 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:28.967060 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:30.528941 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:30.528913 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-kvdtw_5efb57d2-5bf0-4799-8e22-3a3066c94cb2/networking-console-plugin/0.log" Jul 12 19:25:36.626701 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:36.626659 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" podUID="47faa66e-e2bc-4670-9c86-a0d5ff7389d5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 12 19:25:37.856940 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:37.856911 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:37.958581 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:37.958547 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:43.913133 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:43.913042 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:44.013474 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:44.013440 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:44.113798 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:44.113768 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:46.626628 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:46.626591 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" podUID="47faa66e-e2bc-4670-9c86-a0d5ff7389d5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 12 19:25:56.627124 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:56.627075 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" podUID="47faa66e-e2bc-4670-9c86-a0d5ff7389d5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 12 19:25:56.627506 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:56.627153 2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" Jul 12 19:25:56.627601 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:56.627584 2564 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"8ee04ff35aa905a5a45fbb285270654cf44a9f541db19506f1c77729f8da394a"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 12 19:25:56.627644 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:56.627620 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" podUID="47faa66e-e2bc-4670-9c86-a0d5ff7389d5" containerName="service-proxy" containerID="cri-o://8ee04ff35aa905a5a45fbb285270654cf44a9f541db19506f1c77729f8da394a" gracePeriod=30 Jul 12 19:25:57.157112 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:57.157078 2564 generic.go:356] "Generic (PLEG): container finished" podID="47faa66e-e2bc-4670-9c86-a0d5ff7389d5" containerID="8ee04ff35aa905a5a45fbb285270654cf44a9f541db19506f1c77729f8da394a" exitCode=2 Jul 12 19:25:57.157112 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:57.157115 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" event={"ID":"47faa66e-e2bc-4670-9c86-a0d5ff7389d5","Type":"ContainerDied","Data":"8ee04ff35aa905a5a45fbb285270654cf44a9f541db19506f1c77729f8da394a"} Jul 12 19:25:57.157329 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:57.157144 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb5b59cbd-fh49k" event={"ID":"47faa66e-e2bc-4670-9c86-a0d5ff7389d5","Type":"ContainerStarted","Data":"a2551db46f856dd7f11f30ab919e5d9e386df33f8bd6d66e11c15abec9fea72d"} Jul 12 19:25:58.911351 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:58.911309 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:58.958547 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:25:58.958517 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:05.934532 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:05.934491 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs\") pod \"network-metrics-daemon-fz99s\" (UID: \"8a0fd497-acc1-4391-bb37-3a00c2c3362d\") " pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:26:05.936810 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:05.936792 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a0fd497-acc1-4391-bb37-3a00c2c3362d-metrics-certs\") pod \"network-metrics-daemon-fz99s\" (UID: \"8a0fd497-acc1-4391-bb37-3a00c2c3362d\") " pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:26:06.113703 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:06.113674 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-brd5c\"" Jul 12 19:26:06.121642 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:06.121619 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz99s" Jul 12 19:26:06.245793 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:06.245767 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fz99s"] Jul 12 19:26:06.248484 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:26:06.248449 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0fd497_acc1_4391_bb37_3a00c2c3362d.slice/crio-c24755a57f17959af81a901f43ce9561843e08d4c95457d32af083f2ef0da5ac WatchSource:0}: Error finding container c24755a57f17959af81a901f43ce9561843e08d4c95457d32af083f2ef0da5ac: Status 404 returned error can't find the container with id c24755a57f17959af81a901f43ce9561843e08d4c95457d32af083f2ef0da5ac Jul 12 19:26:06.250159 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:06.250142 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 19:26:07.192900 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:07.192874 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fz99s" event={"ID":"8a0fd497-acc1-4391-bb37-3a00c2c3362d","Type":"ContainerStarted","Data":"c24755a57f17959af81a901f43ce9561843e08d4c95457d32af083f2ef0da5ac"} Jul 12 19:26:08.197315 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:08.197276 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fz99s" event={"ID":"8a0fd497-acc1-4391-bb37-3a00c2c3362d","Type":"ContainerStarted","Data":"4c08a9750f41a98ff04b818fd435d720de395f09e8d3766d4dcf74349f587709"} Jul 12 19:26:08.197706 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:08.197329 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fz99s" event={"ID":"8a0fd497-acc1-4391-bb37-3a00c2c3362d","Type":"ContainerStarted","Data":"450e0b8e0e3517c735b45b8b981a70069a414cf5100cf6ec0cd5fbb255ecb2cb"} Jul 12 19:26:08.212559 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:08.212507 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fz99s" podStartSLOduration=375.349549431 podStartE2EDuration="6m16.212494225s" totalImagesPullingTime="862.944794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:26:08.211339186 +0000 UTC m=+376.804613285" watchObservedRunningTime="2026-07-12 19:26:08.212494225 +0000 UTC m=+376.805768323" Jul 12 19:26:08.291664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:08.291617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:08.340722 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:08.340683 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:08.624691 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:08.624600 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:08.724883 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:08.724841 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:16.486339 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:16.486280 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:16.545228 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:16.545188 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:18.394812 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:18.394779 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:18.442630 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:18.442587 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:19.146518 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:19.146479 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:19.198179 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:19.198143 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:30.480673 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:30.480632 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:30.536462 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:30.536431 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:30.585526 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:30.585477 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:52.999796 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:52.999754 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:53.048190 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:26:53.048158 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:23.833310 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:27:23.833273 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:23.936952 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:27:23.936913 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:35.688096 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:27:35.688050 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:35.737873 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:27:35.737843 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:41.908261 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:27:41.908201 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:41.965804 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:27:41.965774 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:43.918613 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:27:43.918576 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:44.019174 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:27:44.019134 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:44.264633 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:27:44.264535 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:52.790980 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:27:52.790927 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:52.842143 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:27:52.842113 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:52.991072 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:27:52.991041 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:12.149668 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:12.149614 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:12.248144 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:12.248111 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:17.932920 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:17.932871 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:33.710896 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:33.710856 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:34.511479 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:34.511446 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:37.308322 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:37.308292 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:37.359072 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:37.359014 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:38.848757 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:38.848720 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:38.949490 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:38.949459 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:39.258579 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:39.258541 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:39.390212 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:39.390171 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:39.812276 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:39.812246 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd"] Jul 12 19:28:39.812485 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:39.812471 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="418ab7c7-cad9-4e69-bde5-d16e643b12d7" containerName="registry" Jul 12 19:28:39.812530 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:39.812488 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="418ab7c7-cad9-4e69-bde5-d16e643b12d7" containerName="registry" Jul 12 19:28:39.812563 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:39.812542 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="418ab7c7-cad9-4e69-bde5-d16e643b12d7" containerName="registry" Jul 12 19:28:39.815404 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:39.815388 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd" Jul 12 19:28:39.817809 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:39.817786 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 12 19:28:39.817928 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:39.817844 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 12 19:28:39.818682 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:39.818669 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sg447\"" Jul 12 19:28:39.824596 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:39.824575 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd"] Jul 12 19:28:39.977575 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:39.977547 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ee5fec3-8583-422f-8336-897cfe86757e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd\" (UID: \"3ee5fec3-8583-422f-8336-897cfe86757e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd" Jul 12 19:28:39.977944 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:39.977580 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v29b9\" (UniqueName: \"kubernetes.io/projected/3ee5fec3-8583-422f-8336-897cfe86757e-kube-api-access-v29b9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd\" (UID: \"3ee5fec3-8583-422f-8336-897cfe86757e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd" Jul 12 19:28:39.977944 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:39.977599 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ee5fec3-8583-422f-8336-897cfe86757e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd\" (UID: \"3ee5fec3-8583-422f-8336-897cfe86757e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd" Jul 12 19:28:40.078618 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:40.078532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ee5fec3-8583-422f-8336-897cfe86757e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd\" (UID: \"3ee5fec3-8583-422f-8336-897cfe86757e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd" Jul 12 19:28:40.078618 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:40.078581 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v29b9\" (UniqueName: \"kubernetes.io/projected/3ee5fec3-8583-422f-8336-897cfe86757e-kube-api-access-v29b9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd\" (UID: \"3ee5fec3-8583-422f-8336-897cfe86757e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd" Jul 12 19:28:40.078618 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:40.078601 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ee5fec3-8583-422f-8336-897cfe86757e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd\" (UID: \"3ee5fec3-8583-422f-8336-897cfe86757e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd" Jul 12 19:28:40.079005 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:40.078974 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ee5fec3-8583-422f-8336-897cfe86757e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd\" (UID: \"3ee5fec3-8583-422f-8336-897cfe86757e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd" Jul 12 19:28:40.079150 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:40.079059 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ee5fec3-8583-422f-8336-897cfe86757e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd\" (UID: \"3ee5fec3-8583-422f-8336-897cfe86757e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd" Jul 12 19:28:40.087618 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:40.087591 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v29b9\" (UniqueName: \"kubernetes.io/projected/3ee5fec3-8583-422f-8336-897cfe86757e-kube-api-access-v29b9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd\" (UID: \"3ee5fec3-8583-422f-8336-897cfe86757e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd" Jul 12 19:28:40.098183 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:40.098141 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:40.124738 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:40.124714 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd" Jul 12 19:28:40.252917 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:40.252891 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd"] Jul 12 19:28:40.255383 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:28:40.255348 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee5fec3_8583_422f_8336_897cfe86757e.slice/crio-992095a26d861fc346eaab2c563dcfb24abf844ecad6b7b1ad5176117e4bb8fb WatchSource:0}: Error finding container 992095a26d861fc346eaab2c563dcfb24abf844ecad6b7b1ad5176117e4bb8fb: Status 404 returned error can't find the container with id 992095a26d861fc346eaab2c563dcfb24abf844ecad6b7b1ad5176117e4bb8fb Jul 12 19:28:40.686737 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:40.686690 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd" event={"ID":"3ee5fec3-8583-422f-8336-897cfe86757e","Type":"ContainerStarted","Data":"992095a26d861fc346eaab2c563dcfb24abf844ecad6b7b1ad5176117e4bb8fb"} Jul 12 19:28:41.118419 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:41.118325 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:41.663804 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:41.663769 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:44.313109 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:44.313066 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:44.371580 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:44.371544 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:46.709108 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:46.709075 2564 generic.go:356] "Generic (PLEG): container finished" podID="3ee5fec3-8583-422f-8336-897cfe86757e" containerID="70892a7309c51a71755e2e7e9806e67a0a03dcb29653c829bac755ea0696e911" exitCode=0 Jul 12 19:28:46.709493 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:46.709160 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd" event={"ID":"3ee5fec3-8583-422f-8336-897cfe86757e","Type":"ContainerDied","Data":"70892a7309c51a71755e2e7e9806e67a0a03dcb29653c829bac755ea0696e911"} Jul 12 19:28:48.443610 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:48.443566 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:48.493284 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:48.493246 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:49.455556 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:49.455505 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:49.723283 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:49.721712 2564 generic.go:356] "Generic (PLEG): container finished" podID="3ee5fec3-8583-422f-8336-897cfe86757e" containerID="242148279cff9487727ae286aaf7e2e01360fc4b9266e77ae2c8e22d38ce6ef7" exitCode=0 Jul 12 19:28:49.723283 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:49.721795 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd" event={"ID":"3ee5fec3-8583-422f-8336-897cfe86757e","Type":"ContainerDied","Data":"242148279cff9487727ae286aaf7e2e01360fc4b9266e77ae2c8e22d38ce6ef7"} Jul 12 19:28:56.752834 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:56.752794 2564 generic.go:356] "Generic (PLEG): container finished" podID="3ee5fec3-8583-422f-8336-897cfe86757e" containerID="c3eab4f24861bbb7ce7cf592618d6dfa5e3192523659349edce2e85211cc6950" exitCode=0 Jul 12 19:28:56.753236 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:56.752874 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd" event={"ID":"3ee5fec3-8583-422f-8336-897cfe86757e","Type":"ContainerDied","Data":"c3eab4f24861bbb7ce7cf592618d6dfa5e3192523659349edce2e85211cc6950"} Jul 12 19:28:57.878305 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:57.878280 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd" Jul 12 19:28:57.904708 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:57.904684 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ee5fec3-8583-422f-8336-897cfe86757e-bundle\") pod \"3ee5fec3-8583-422f-8336-897cfe86757e\" (UID: \"3ee5fec3-8583-422f-8336-897cfe86757e\") " Jul 12 19:28:57.904833 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:57.904751 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v29b9\" (UniqueName: \"kubernetes.io/projected/3ee5fec3-8583-422f-8336-897cfe86757e-kube-api-access-v29b9\") pod \"3ee5fec3-8583-422f-8336-897cfe86757e\" (UID: \"3ee5fec3-8583-422f-8336-897cfe86757e\") " Jul 12 19:28:57.904833 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:57.904806 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ee5fec3-8583-422f-8336-897cfe86757e-util\") pod \"3ee5fec3-8583-422f-8336-897cfe86757e\" (UID: \"3ee5fec3-8583-422f-8336-897cfe86757e\") " Jul 12 19:28:57.905141 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:57.905117 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee5fec3-8583-422f-8336-897cfe86757e-bundle" (OuterVolumeSpecName: "bundle") pod "3ee5fec3-8583-422f-8336-897cfe86757e" (UID: "3ee5fec3-8583-422f-8336-897cfe86757e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:28:57.906991 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:57.906967 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee5fec3-8583-422f-8336-897cfe86757e-kube-api-access-v29b9" (OuterVolumeSpecName: "kube-api-access-v29b9") pod "3ee5fec3-8583-422f-8336-897cfe86757e" (UID: "3ee5fec3-8583-422f-8336-897cfe86757e"). InnerVolumeSpecName "kube-api-access-v29b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:28:57.908922 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:57.908892 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee5fec3-8583-422f-8336-897cfe86757e-util" (OuterVolumeSpecName: "util") pod "3ee5fec3-8583-422f-8336-897cfe86757e" (UID: "3ee5fec3-8583-422f-8336-897cfe86757e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:28:58.005318 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:58.005282 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ee5fec3-8583-422f-8336-897cfe86757e-util\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:28:58.005318 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:58.005304 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ee5fec3-8583-422f-8336-897cfe86757e-bundle\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:28:58.005318 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:58.005313 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v29b9\" (UniqueName: \"kubernetes.io/projected/3ee5fec3-8583-422f-8336-897cfe86757e-kube-api-access-v29b9\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:28:58.761324 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:58.761290 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd" event={"ID":"3ee5fec3-8583-422f-8336-897cfe86757e","Type":"ContainerDied","Data":"992095a26d861fc346eaab2c563dcfb24abf844ecad6b7b1ad5176117e4bb8fb"} Jul 12 19:28:58.761324 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:58.761307 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f574sd" Jul 12 19:28:58.761530 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:28:58.761335 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="992095a26d861fc346eaab2c563dcfb24abf844ecad6b7b1ad5176117e4bb8fb" Jul 12 19:29:02.194389 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:02.194357 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:02.287003 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:02.286963 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:04.216177 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.216143 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkh4l"] Jul 12 19:29:04.216564 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.216382 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ee5fec3-8583-422f-8336-897cfe86757e" containerName="pull" Jul 12 19:29:04.216564 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.216393 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ee5fec3-8583-422f-8336-897cfe86757e" containerName="pull" Jul 12 19:29:04.216564 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.216401 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ee5fec3-8583-422f-8336-897cfe86757e" containerName="util" Jul 12 19:29:04.216564 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.216406 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ee5fec3-8583-422f-8336-897cfe86757e" containerName="util" Jul 12 19:29:04.216564 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.216412 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ee5fec3-8583-422f-8336-897cfe86757e" containerName="extract" Jul 12 19:29:04.216564 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.216417 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ee5fec3-8583-422f-8336-897cfe86757e" containerName="extract" Jul 12 19:29:04.216564 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.216463 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ee5fec3-8583-422f-8336-897cfe86757e" containerName="extract" Jul 12 19:29:04.218478 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.218463 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkh4l" Jul 12 19:29:04.220901 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.220878 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 12 19:29:04.220901 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.220895 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 12 19:29:04.221851 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.221833 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-d5pk8\"" Jul 12 19:29:04.233284 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.233262 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkh4l"] Jul 12 19:29:04.246491 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.246468 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d39fc2e8-d1dd-4f6f-99f8-7fd4c1b66b81-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gkh4l\" (UID: \"d39fc2e8-d1dd-4f6f-99f8-7fd4c1b66b81\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkh4l" Jul 12 19:29:04.246596 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.246526 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq7tz\" (UniqueName: \"kubernetes.io/projected/d39fc2e8-d1dd-4f6f-99f8-7fd4c1b66b81-kube-api-access-sq7tz\") pod \"openshift-lws-operator-fd8ccff4c-gkh4l\" (UID: \"d39fc2e8-d1dd-4f6f-99f8-7fd4c1b66b81\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkh4l" Jul 12 19:29:04.347180 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.347146 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq7tz\" (UniqueName: \"kubernetes.io/projected/d39fc2e8-d1dd-4f6f-99f8-7fd4c1b66b81-kube-api-access-sq7tz\") pod \"openshift-lws-operator-fd8ccff4c-gkh4l\" (UID: \"d39fc2e8-d1dd-4f6f-99f8-7fd4c1b66b81\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkh4l" Jul 12 19:29:04.347327 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.347200 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d39fc2e8-d1dd-4f6f-99f8-7fd4c1b66b81-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gkh4l\" (UID: \"d39fc2e8-d1dd-4f6f-99f8-7fd4c1b66b81\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkh4l" Jul 12 19:29:04.347558 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.347538 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d39fc2e8-d1dd-4f6f-99f8-7fd4c1b66b81-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gkh4l\" (UID: \"d39fc2e8-d1dd-4f6f-99f8-7fd4c1b66b81\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkh4l" Jul 12 19:29:04.355939 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.355921 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq7tz\" (UniqueName: \"kubernetes.io/projected/d39fc2e8-d1dd-4f6f-99f8-7fd4c1b66b81-kube-api-access-sq7tz\") pod \"openshift-lws-operator-fd8ccff4c-gkh4l\" (UID: \"d39fc2e8-d1dd-4f6f-99f8-7fd4c1b66b81\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkh4l" Jul 12 19:29:04.527676 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.527580 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkh4l" Jul 12 19:29:04.650302 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.650277 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkh4l"] Jul 12 19:29:04.652760 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:29:04.652729 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd39fc2e8_d1dd_4f6f_99f8_7fd4c1b66b81.slice/crio-31042315cf23a3b4c998ec7864f39d467275728b6343c3930450cbc7f1e3f6d3 WatchSource:0}: Error finding container 31042315cf23a3b4c998ec7864f39d467275728b6343c3930450cbc7f1e3f6d3: Status 404 returned error can't find the container with id 31042315cf23a3b4c998ec7864f39d467275728b6343c3930450cbc7f1e3f6d3 Jul 12 19:29:04.782251 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:04.782174 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkh4l" event={"ID":"d39fc2e8-d1dd-4f6f-99f8-7fd4c1b66b81","Type":"ContainerStarted","Data":"31042315cf23a3b4c998ec7864f39d467275728b6343c3930450cbc7f1e3f6d3"} Jul 12 19:29:06.791323 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:06.791277 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkh4l" event={"ID":"d39fc2e8-d1dd-4f6f-99f8-7fd4c1b66b81","Type":"ContainerStarted","Data":"23df36aaaca89f2e933b6221130a863aa9fc8efd5f562de796c9b463fb070198"} Jul 12 19:29:06.806414 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:06.806363 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkh4l" podStartSLOduration=1.220830394 podStartE2EDuration="2.806349554s" totalImagesPullingTime="1.58551916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:29:06.80528191 +0000 UTC m=+555.398556008" watchObservedRunningTime="2026-07-12 19:29:06.806349554 +0000 UTC m=+555.399623651" Jul 12 19:29:07.500975 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:07.500941 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:07.587282 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:07.587249 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:09.273795 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:09.273760 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:13.481672 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:13.481635 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:13.555943 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:13.555903 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:15.588787 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:15.588753 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:15.640382 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:15.640337 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:22.598969 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:22.598932 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:23.386904 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:23.386865 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:23.486216 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:23.486175 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:24.194862 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.194821 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:24.452012 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.451917 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:24.526894 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.526860 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:24.531842 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.531814 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g"] Jul 12 19:29:24.536461 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.536444 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" Jul 12 19:29:24.539136 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.539110 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 12 19:29:24.539136 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.539123 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 12 19:29:24.539404 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.539389 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 12 19:29:24.539471 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.539391 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 12 19:29:24.539615 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.539603 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4cqmv\"" Jul 12 19:29:24.550822 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.550799 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g"] Jul 12 19:29:24.584620 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.584591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/393fd51b-4d9e-4f83-9eac-82d93b1d7d69-manifests\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-vgj7g\" (UID: \"393fd51b-4d9e-4f83-9eac-82d93b1d7d69\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" Jul 12 19:29:24.584620 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.584618 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/393fd51b-4d9e-4f83-9eac-82d93b1d7d69-tmp\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-vgj7g\" (UID: \"393fd51b-4d9e-4f83-9eac-82d93b1d7d69\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" Jul 12 19:29:24.584783 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.584637 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/393fd51b-4d9e-4f83-9eac-82d93b1d7d69-webhook-cert\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-vgj7g\" (UID: \"393fd51b-4d9e-4f83-9eac-82d93b1d7d69\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" Jul 12 19:29:24.584783 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.584686 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2j2r\" (UniqueName: \"kubernetes.io/projected/393fd51b-4d9e-4f83-9eac-82d93b1d7d69-kube-api-access-b2j2r\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-vgj7g\" (UID: \"393fd51b-4d9e-4f83-9eac-82d93b1d7d69\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" Jul 12 19:29:24.584783 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.584714 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/393fd51b-4d9e-4f83-9eac-82d93b1d7d69-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-vgj7g\" (UID: \"393fd51b-4d9e-4f83-9eac-82d93b1d7d69\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" Jul 12 19:29:24.678385 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.678347 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:24.685272 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.685251 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2j2r\" (UniqueName: \"kubernetes.io/projected/393fd51b-4d9e-4f83-9eac-82d93b1d7d69-kube-api-access-b2j2r\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-vgj7g\" (UID: \"393fd51b-4d9e-4f83-9eac-82d93b1d7d69\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" Jul 12 19:29:24.685364 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.685286 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/393fd51b-4d9e-4f83-9eac-82d93b1d7d69-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-vgj7g\" (UID: \"393fd51b-4d9e-4f83-9eac-82d93b1d7d69\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" Jul 12 19:29:24.685364 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.685314 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/393fd51b-4d9e-4f83-9eac-82d93b1d7d69-manifests\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-vgj7g\" (UID: \"393fd51b-4d9e-4f83-9eac-82d93b1d7d69\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" Jul 12 19:29:24.685364 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.685331 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/393fd51b-4d9e-4f83-9eac-82d93b1d7d69-tmp\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-vgj7g\" (UID: \"393fd51b-4d9e-4f83-9eac-82d93b1d7d69\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" Jul 12 19:29:24.685364 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.685350 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/393fd51b-4d9e-4f83-9eac-82d93b1d7d69-webhook-cert\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-vgj7g\" (UID: \"393fd51b-4d9e-4f83-9eac-82d93b1d7d69\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" Jul 12 19:29:24.685677 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.685660 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/393fd51b-4d9e-4f83-9eac-82d93b1d7d69-manifests\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-vgj7g\" (UID: \"393fd51b-4d9e-4f83-9eac-82d93b1d7d69\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" Jul 12 19:29:24.687882 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.687846 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/393fd51b-4d9e-4f83-9eac-82d93b1d7d69-tmp\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-vgj7g\" (UID: \"393fd51b-4d9e-4f83-9eac-82d93b1d7d69\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" Jul 12 19:29:24.687981 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.687951 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/393fd51b-4d9e-4f83-9eac-82d93b1d7d69-webhook-cert\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-vgj7g\" (UID: \"393fd51b-4d9e-4f83-9eac-82d93b1d7d69\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" Jul 12 19:29:24.688072 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.687985 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/393fd51b-4d9e-4f83-9eac-82d93b1d7d69-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-vgj7g\" (UID: \"393fd51b-4d9e-4f83-9eac-82d93b1d7d69\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" Jul 12 19:29:24.693277 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.693249 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2j2r\" (UniqueName: \"kubernetes.io/projected/393fd51b-4d9e-4f83-9eac-82d93b1d7d69-kube-api-access-b2j2r\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-vgj7g\" (UID: \"393fd51b-4d9e-4f83-9eac-82d93b1d7d69\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" Jul 12 19:29:24.785838 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.785748 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:24.847183 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.847154 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" Jul 12 19:29:24.879414 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.879378 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:24.977315 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.977289 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g"] Jul 12 19:29:24.979872 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:29:24.979843 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393fd51b_4d9e_4f83_9eac_82d93b1d7d69.slice/crio-d06e40435afdc5a71537578b8789c117fd6c48b01d6e9b274f2ae6677f1c319b WatchSource:0}: Error finding container d06e40435afdc5a71537578b8789c117fd6c48b01d6e9b274f2ae6677f1c319b: Status 404 returned error can't find the container with id d06e40435afdc5a71537578b8789c117fd6c48b01d6e9b274f2ae6677f1c319b Jul 12 19:29:24.983859 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:24.983831 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:25.084724 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:25.084632 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:25.858636 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:25.858597 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" event={"ID":"393fd51b-4d9e-4f83-9eac-82d93b1d7d69","Type":"ContainerStarted","Data":"d06e40435afdc5a71537578b8789c117fd6c48b01d6e9b274f2ae6677f1c319b"} Jul 12 19:29:28.872128 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:28.872046 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" event={"ID":"393fd51b-4d9e-4f83-9eac-82d93b1d7d69","Type":"ContainerStarted","Data":"983a599ac26619d4b6f96452d5dee1d81ff0e40ec70ca5b2d14903d3811be965"} Jul 12 19:29:29.876786 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:29.876691 2564 generic.go:356] "Generic (PLEG): container finished" podID="393fd51b-4d9e-4f83-9eac-82d93b1d7d69" containerID="983a599ac26619d4b6f96452d5dee1d81ff0e40ec70ca5b2d14903d3811be965" exitCode=0 Jul 12 19:29:29.877227 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:29.876780 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" event={"ID":"393fd51b-4d9e-4f83-9eac-82d93b1d7d69","Type":"ContainerDied","Data":"983a599ac26619d4b6f96452d5dee1d81ff0e40ec70ca5b2d14903d3811be965"} Jul 12 19:29:30.882486 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:30.882442 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" event={"ID":"393fd51b-4d9e-4f83-9eac-82d93b1d7d69","Type":"ContainerStarted","Data":"f06797f3f832fb5ae99b361a3636d0ef0ee2b2b3be8f7b3553b700cd3132f7ae"} Jul 12 19:29:30.882890 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:30.882567 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" Jul 12 19:29:30.910860 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:30.910806 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" podStartSLOduration=2.079993313 podStartE2EDuration="6.910791477s" totalImagesPullingTime="3.843943167s" totalInitContainerRuntime="986.854997ms" isStatefulPod=false podCreationTimestamp="2026-07-12 19:29:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:29:30.908660252 +0000 UTC m=+579.501934390" watchObservedRunningTime="2026-07-12 19:29:30.910791477 +0000 UTC m=+579.504065575" Jul 12 19:29:36.601187 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:36.601149 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:36.701332 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:36.701294 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:41.888043 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:41.887990 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-vgj7g" Jul 12 19:29:43.618386 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:43.618347 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:43.710530 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:43.710491 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:44.030209 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.030171 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:44.159275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.159238 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-59ff7fb449-z4k9f"] Jul 12 19:29:44.162520 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.162499 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59ff7fb449-z4k9f" Jul 12 19:29:44.165058 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.165000 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 12 19:29:44.165165 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.165132 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 12 19:29:44.166177 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.166157 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 12 19:29:44.166240 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.166205 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 12 19:29:44.166240 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.166161 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wqx6k\"" Jul 12 19:29:44.172327 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.172306 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59ff7fb449-z4k9f"] Jul 12 19:29:44.228298 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.228271 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d20bc6a-a256-4490-951e-e38d805189b7-tls-certs\") pod \"kube-auth-proxy-59ff7fb449-z4k9f\" (UID: \"3d20bc6a-a256-4490-951e-e38d805189b7\") " pod="openshift-ingress/kube-auth-proxy-59ff7fb449-z4k9f" Jul 12 19:29:44.228424 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.228330 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d20bc6a-a256-4490-951e-e38d805189b7-tmp\") pod \"kube-auth-proxy-59ff7fb449-z4k9f\" (UID: \"3d20bc6a-a256-4490-951e-e38d805189b7\") " pod="openshift-ingress/kube-auth-proxy-59ff7fb449-z4k9f" Jul 12 19:29:44.228424 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.228364 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l478k\" (UniqueName: \"kubernetes.io/projected/3d20bc6a-a256-4490-951e-e38d805189b7-kube-api-access-l478k\") pod \"kube-auth-proxy-59ff7fb449-z4k9f\" (UID: \"3d20bc6a-a256-4490-951e-e38d805189b7\") " pod="openshift-ingress/kube-auth-proxy-59ff7fb449-z4k9f" Jul 12 19:29:44.254708 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.254678 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bm6ks"] Jul 12 19:29:44.257625 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.257604 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-bm6ks" Jul 12 19:29:44.260348 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.260325 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 12 19:29:44.260722 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.260706 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-bdb4m\"" Jul 12 19:29:44.274368 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.274339 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bm6ks"] Jul 12 19:29:44.295419 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.295338 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:44.328943 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.328915 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6133c191-d1e4-4476-b2c8-aeee1b3d7c4e-cert\") pod \"odh-model-controller-7998c67dc4-bm6ks\" (UID: \"6133c191-d1e4-4476-b2c8-aeee1b3d7c4e\") " pod="opendatahub/odh-model-controller-7998c67dc4-bm6ks" Jul 12 19:29:44.328943 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.328947 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx92c\" (UniqueName: \"kubernetes.io/projected/6133c191-d1e4-4476-b2c8-aeee1b3d7c4e-kube-api-access-nx92c\") pod \"odh-model-controller-7998c67dc4-bm6ks\" (UID: \"6133c191-d1e4-4476-b2c8-aeee1b3d7c4e\") " pod="opendatahub/odh-model-controller-7998c67dc4-bm6ks" Jul 12 19:29:44.329123 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.328973 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d20bc6a-a256-4490-951e-e38d805189b7-tmp\") pod \"kube-auth-proxy-59ff7fb449-z4k9f\" (UID: \"3d20bc6a-a256-4490-951e-e38d805189b7\") " pod="openshift-ingress/kube-auth-proxy-59ff7fb449-z4k9f" Jul 12 19:29:44.329123 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.329000 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l478k\" (UniqueName: \"kubernetes.io/projected/3d20bc6a-a256-4490-951e-e38d805189b7-kube-api-access-l478k\") pod \"kube-auth-proxy-59ff7fb449-z4k9f\" (UID: \"3d20bc6a-a256-4490-951e-e38d805189b7\") " pod="openshift-ingress/kube-auth-proxy-59ff7fb449-z4k9f" Jul 12 19:29:44.329123 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.329063 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d20bc6a-a256-4490-951e-e38d805189b7-tls-certs\") pod \"kube-auth-proxy-59ff7fb449-z4k9f\" (UID: \"3d20bc6a-a256-4490-951e-e38d805189b7\") " pod="openshift-ingress/kube-auth-proxy-59ff7fb449-z4k9f" Jul 12 19:29:44.329218 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:29:44.329157 2564 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 12 19:29:44.329260 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:29:44.329229 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d20bc6a-a256-4490-951e-e38d805189b7-tls-certs podName:3d20bc6a-a256-4490-951e-e38d805189b7 nodeName:}" failed. No retries permitted until 2026-07-12 19:29:44.829211017 +0000 UTC m=+593.422485092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3d20bc6a-a256-4490-951e-e38d805189b7-tls-certs") pod "kube-auth-proxy-59ff7fb449-z4k9f" (UID: "3d20bc6a-a256-4490-951e-e38d805189b7") : secret "kube-auth-proxy-tls" not found Jul 12 19:29:44.331209 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.331188 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d20bc6a-a256-4490-951e-e38d805189b7-tmp\") pod \"kube-auth-proxy-59ff7fb449-z4k9f\" (UID: \"3d20bc6a-a256-4490-951e-e38d805189b7\") " pod="openshift-ingress/kube-auth-proxy-59ff7fb449-z4k9f" Jul 12 19:29:44.341156 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.341128 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l478k\" (UniqueName: \"kubernetes.io/projected/3d20bc6a-a256-4490-951e-e38d805189b7-kube-api-access-l478k\") pod \"kube-auth-proxy-59ff7fb449-z4k9f\" (UID: \"3d20bc6a-a256-4490-951e-e38d805189b7\") " pod="openshift-ingress/kube-auth-proxy-59ff7fb449-z4k9f" Jul 12 19:29:44.391405 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.391355 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:44.429992 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.429956 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6133c191-d1e4-4476-b2c8-aeee1b3d7c4e-cert\") pod \"odh-model-controller-7998c67dc4-bm6ks\" (UID: \"6133c191-d1e4-4476-b2c8-aeee1b3d7c4e\") " pod="opendatahub/odh-model-controller-7998c67dc4-bm6ks" Jul 12 19:29:44.430164 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.430005 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx92c\" (UniqueName: \"kubernetes.io/projected/6133c191-d1e4-4476-b2c8-aeee1b3d7c4e-kube-api-access-nx92c\") pod \"odh-model-controller-7998c67dc4-bm6ks\" (UID: \"6133c191-d1e4-4476-b2c8-aeee1b3d7c4e\") " pod="opendatahub/odh-model-controller-7998c67dc4-bm6ks" Jul 12 19:29:44.430164 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:29:44.430135 2564 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 12 19:29:44.430253 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:29:44.430213 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6133c191-d1e4-4476-b2c8-aeee1b3d7c4e-cert podName:6133c191-d1e4-4476-b2c8-aeee1b3d7c4e nodeName:}" failed. No retries permitted until 2026-07-12 19:29:44.930190569 +0000 UTC m=+593.523464645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6133c191-d1e4-4476-b2c8-aeee1b3d7c4e-cert") pod "odh-model-controller-7998c67dc4-bm6ks" (UID: "6133c191-d1e4-4476-b2c8-aeee1b3d7c4e") : secret "odh-model-controller-webhook-cert" not found Jul 12 19:29:44.439323 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.439289 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx92c\" (UniqueName: \"kubernetes.io/projected/6133c191-d1e4-4476-b2c8-aeee1b3d7c4e-kube-api-access-nx92c\") pod \"odh-model-controller-7998c67dc4-bm6ks\" (UID: \"6133c191-d1e4-4476-b2c8-aeee1b3d7c4e\") " pod="opendatahub/odh-model-controller-7998c67dc4-bm6ks" Jul 12 19:29:44.487309 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.487271 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:44.625373 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.625255 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:44.834577 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.834547 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d20bc6a-a256-4490-951e-e38d805189b7-tls-certs\") pod \"kube-auth-proxy-59ff7fb449-z4k9f\" (UID: \"3d20bc6a-a256-4490-951e-e38d805189b7\") " pod="openshift-ingress/kube-auth-proxy-59ff7fb449-z4k9f" Jul 12 19:29:44.837400 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.837364 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d20bc6a-a256-4490-951e-e38d805189b7-tls-certs\") pod \"kube-auth-proxy-59ff7fb449-z4k9f\" (UID: \"3d20bc6a-a256-4490-951e-e38d805189b7\") " pod="openshift-ingress/kube-auth-proxy-59ff7fb449-z4k9f" Jul 12 19:29:44.842981 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.842954 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:44.935206 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.935176 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6133c191-d1e4-4476-b2c8-aeee1b3d7c4e-cert\") pod \"odh-model-controller-7998c67dc4-bm6ks\" (UID: \"6133c191-d1e4-4476-b2c8-aeee1b3d7c4e\") " pod="opendatahub/odh-model-controller-7998c67dc4-bm6ks" Jul 12 19:29:44.937649 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:44.937619 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6133c191-d1e4-4476-b2c8-aeee1b3d7c4e-cert\") pod \"odh-model-controller-7998c67dc4-bm6ks\" (UID: \"6133c191-d1e4-4476-b2c8-aeee1b3d7c4e\") " pod="opendatahub/odh-model-controller-7998c67dc4-bm6ks" Jul 12 19:29:45.014194 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.014154 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:45.067163 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.067121 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zkrk2"] Jul 12 19:29:45.071656 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.071632 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59ff7fb449-z4k9f" Jul 12 19:29:45.071769 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.071632 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zkrk2" Jul 12 19:29:45.074040 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.073999 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 12 19:29:45.074171 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.074108 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-c7lh2\"" Jul 12 19:29:45.080771 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.080746 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zkrk2"] Jul 12 19:29:45.136898 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.136868 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f65c6d-a6bf-4977-a40e-704613ac348b-tls-certs\") pod \"model-serving-api-54d89bd5fc-zkrk2\" (UID: \"a5f65c6d-a6bf-4977-a40e-704613ac348b\") " pod="opendatahub/model-serving-api-54d89bd5fc-zkrk2" Jul 12 19:29:45.137128 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.136910 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzv2n\" (UniqueName: \"kubernetes.io/projected/a5f65c6d-a6bf-4977-a40e-704613ac348b-kube-api-access-wzv2n\") pod \"model-serving-api-54d89bd5fc-zkrk2\" (UID: \"a5f65c6d-a6bf-4977-a40e-704613ac348b\") " pod="opendatahub/model-serving-api-54d89bd5fc-zkrk2" Jul 12 19:29:45.167338 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.167309 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-bm6ks" Jul 12 19:29:45.215171 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.215140 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59ff7fb449-z4k9f"] Jul 12 19:29:45.217626 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:29:45.217600 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d20bc6a_a256_4490_951e_e38d805189b7.slice/crio-77ea3eea1815254fce830ebd426fb0abe70241961c946ad5565be7663bd0790e WatchSource:0}: Error finding container 77ea3eea1815254fce830ebd426fb0abe70241961c946ad5565be7663bd0790e: Status 404 returned error can't find the container with id 77ea3eea1815254fce830ebd426fb0abe70241961c946ad5565be7663bd0790e Jul 12 19:29:45.241998 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.241330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f65c6d-a6bf-4977-a40e-704613ac348b-tls-certs\") pod \"model-serving-api-54d89bd5fc-zkrk2\" (UID: \"a5f65c6d-a6bf-4977-a40e-704613ac348b\") " pod="opendatahub/model-serving-api-54d89bd5fc-zkrk2" Jul 12 19:29:45.241998 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.241376 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzv2n\" (UniqueName: \"kubernetes.io/projected/a5f65c6d-a6bf-4977-a40e-704613ac348b-kube-api-access-wzv2n\") pod \"model-serving-api-54d89bd5fc-zkrk2\" (UID: \"a5f65c6d-a6bf-4977-a40e-704613ac348b\") " pod="opendatahub/model-serving-api-54d89bd5fc-zkrk2" Jul 12 19:29:45.241998 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:29:45.241570 2564 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 12 19:29:45.241998 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:29:45.241635 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5f65c6d-a6bf-4977-a40e-704613ac348b-tls-certs podName:a5f65c6d-a6bf-4977-a40e-704613ac348b nodeName:}" failed. No retries permitted until 2026-07-12 19:29:45.741614234 +0000 UTC m=+594.334888313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a5f65c6d-a6bf-4977-a40e-704613ac348b-tls-certs") pod "model-serving-api-54d89bd5fc-zkrk2" (UID: "a5f65c6d-a6bf-4977-a40e-704613ac348b") : secret "model-serving-api-tls" not found Jul 12 19:29:45.252902 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.252845 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzv2n\" (UniqueName: \"kubernetes.io/projected/a5f65c6d-a6bf-4977-a40e-704613ac348b-kube-api-access-wzv2n\") pod \"model-serving-api-54d89bd5fc-zkrk2\" (UID: \"a5f65c6d-a6bf-4977-a40e-704613ac348b\") " pod="opendatahub/model-serving-api-54d89bd5fc-zkrk2" Jul 12 19:29:45.332458 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.332416 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bm6ks"] Jul 12 19:29:45.334979 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:29:45.334953 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6133c191_d1e4_4476_b2c8_aeee1b3d7c4e.slice/crio-b987d0e3fe823011899abb85ca54f42854f23b1d1ee155d389fcf7f17d3bc8bf WatchSource:0}: Error finding container b987d0e3fe823011899abb85ca54f42854f23b1d1ee155d389fcf7f17d3bc8bf: Status 404 returned error can't find the container with id b987d0e3fe823011899abb85ca54f42854f23b1d1ee155d389fcf7f17d3bc8bf Jul 12 19:29:45.413617 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.413583 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-z58j6"] Jul 12 19:29:45.418097 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.418076 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.420636 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.420608 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 12 19:29:45.420636 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.420625 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 12 19:29:45.420839 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.420626 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 12 19:29:45.420839 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.420608 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 12 19:29:45.420839 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.420755 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-hxpsr\"" Jul 12 19:29:45.430102 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.430078 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-z58j6"] Jul 12 19:29:45.433710 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.433687 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:45.495439 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.495353 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:45.544170 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.544142 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/958bd3cb-cfc7-430c-b79d-1542b6d34e24-local-certs\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.544170 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.544171 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/958bd3cb-cfc7-430c-b79d-1542b6d34e24-cacerts\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.544337 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.544186 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.544337 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.544245 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-token\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.544337 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.544281 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.544427 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.544344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff94t\" (UniqueName: \"kubernetes.io/projected/958bd3cb-cfc7-430c-b79d-1542b6d34e24-kube-api-access-ff94t\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.544427 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.544366 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.645301 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.645261 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.645795 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.645357 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/958bd3cb-cfc7-430c-b79d-1542b6d34e24-local-certs\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.645795 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.645375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/958bd3cb-cfc7-430c-b79d-1542b6d34e24-cacerts\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.645795 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.645394 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.645795 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.645425 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-token\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.645795 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.645453 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.645795 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.645546 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff94t\" (UniqueName: \"kubernetes.io/projected/958bd3cb-cfc7-430c-b79d-1542b6d34e24-kube-api-access-ff94t\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.646417 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.646358 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.648582 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.648557 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.648680 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.648637 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/958bd3cb-cfc7-430c-b79d-1542b6d34e24-cacerts\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.648801 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.648780 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.648859 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.648822 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/958bd3cb-cfc7-430c-b79d-1542b6d34e24-local-certs\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.653145 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.653125 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-token\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.653230 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.653184 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff94t\" (UniqueName: \"kubernetes.io/projected/958bd3cb-cfc7-430c-b79d-1542b6d34e24-kube-api-access-ff94t\") pod \"istiod-openshift-gateway-7574459656-z58j6\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.727612 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.727571 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:45.746703 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.746599 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f65c6d-a6bf-4977-a40e-704613ac348b-tls-certs\") pod \"model-serving-api-54d89bd5fc-zkrk2\" (UID: \"a5f65c6d-a6bf-4977-a40e-704613ac348b\") " pod="opendatahub/model-serving-api-54d89bd5fc-zkrk2" Jul 12 19:29:45.749060 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.749039 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f65c6d-a6bf-4977-a40e-704613ac348b-tls-certs\") pod \"model-serving-api-54d89bd5fc-zkrk2\" (UID: \"a5f65c6d-a6bf-4977-a40e-704613ac348b\") " pod="opendatahub/model-serving-api-54d89bd5fc-zkrk2" Jul 12 19:29:45.751745 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.751706 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:45.865138 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.865100 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-z58j6"] Jul 12 19:29:45.869074 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:29:45.869037 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958bd3cb_cfc7_430c_b79d_1542b6d34e24.slice/crio-8adeb08c8f1e7e35383235b8ef90956792fefe8da547d7716754038a61a36d7b WatchSource:0}: Error finding container 8adeb08c8f1e7e35383235b8ef90956792fefe8da547d7716754038a61a36d7b: Status 404 returned error can't find the container with id 8adeb08c8f1e7e35383235b8ef90956792fefe8da547d7716754038a61a36d7b Jul 12 19:29:45.938814 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.937608 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" event={"ID":"958bd3cb-cfc7-430c-b79d-1542b6d34e24","Type":"ContainerStarted","Data":"8adeb08c8f1e7e35383235b8ef90956792fefe8da547d7716754038a61a36d7b"} Jul 12 19:29:45.939642 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.939610 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59ff7fb449-z4k9f" event={"ID":"3d20bc6a-a256-4490-951e-e38d805189b7","Type":"ContainerStarted","Data":"77ea3eea1815254fce830ebd426fb0abe70241961c946ad5565be7663bd0790e"} Jul 12 19:29:45.941259 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.941194 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-bm6ks" event={"ID":"6133c191-d1e4-4476-b2c8-aeee1b3d7c4e","Type":"ContainerStarted","Data":"b987d0e3fe823011899abb85ca54f42854f23b1d1ee155d389fcf7f17d3bc8bf"} Jul 12 19:29:45.982570 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:45.982538 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zkrk2" Jul 12 19:29:46.151729 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:46.151692 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zkrk2"] Jul 12 19:29:46.156699 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:29:46.156660 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f65c6d_a6bf_4977_a40e_704613ac348b.slice/crio-a8e0c47c751312e3dc431485374aa47b65a076fa9965906a63a78137f9a75d80 WatchSource:0}: Error finding container a8e0c47c751312e3dc431485374aa47b65a076fa9965906a63a78137f9a75d80: Status 404 returned error can't find the container with id a8e0c47c751312e3dc431485374aa47b65a076fa9965906a63a78137f9a75d80 Jul 12 19:29:46.258181 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:46.258116 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:46.949172 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:46.949132 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zkrk2" event={"ID":"a5f65c6d-a6bf-4977-a40e-704613ac348b","Type":"ContainerStarted","Data":"a8e0c47c751312e3dc431485374aa47b65a076fa9965906a63a78137f9a75d80"} Jul 12 19:29:50.840081 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:50.840048 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:29:50.840458 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:50.840108 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:29:50.977119 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:50.977065 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59ff7fb449-z4k9f" event={"ID":"3d20bc6a-a256-4490-951e-e38d805189b7","Type":"ContainerStarted","Data":"7da25ea3f393521bd7622f655978ac74bd9720bd0578352e759884b01dd18cb8"} Jul 12 19:29:50.979188 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:50.979158 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-bm6ks" event={"ID":"6133c191-d1e4-4476-b2c8-aeee1b3d7c4e","Type":"ContainerStarted","Data":"7afc22df780bcc4e1168836514f7cc6df4758c70fb275f0f6e41358c9f8beccc"} Jul 12 19:29:50.979342 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:50.979216 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-bm6ks" Jul 12 19:29:50.980953 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:50.980931 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" event={"ID":"958bd3cb-cfc7-430c-b79d-1542b6d34e24","Type":"ContainerStarted","Data":"014cf2f3c02a6956856878af6249c1c0dc8ccb1bc76e989cccc8d31aeb917269"} Jul 12 19:29:50.981138 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:50.981121 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:50.982546 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:50.982530 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zkrk2" event={"ID":"a5f65c6d-a6bf-4977-a40e-704613ac348b","Type":"ContainerStarted","Data":"a402784a5a7734fdfd51b6ec13e048190338620cb1c7605f898578f1577215cb"} Jul 12 19:29:50.982787 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:50.982774 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-zkrk2" Jul 12 19:29:50.993855 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:50.993802 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-59ff7fb449-z4k9f" podStartSLOduration=1.381056228 podStartE2EDuration="6.993786194s" totalImagesPullingTime="5.612729966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:29:50.991962832 +0000 UTC m=+599.585236931" watchObservedRunningTime="2026-07-12 19:29:50.993786194 +0000 UTC m=+599.587060291" Jul 12 19:29:51.007252 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:51.007190 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-bm6ks" podStartSLOduration=1.4955024190000001 podStartE2EDuration="7.007167241s" totalImagesPullingTime="5.511664822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:29:51.006362592 +0000 UTC m=+599.599636691" watchObservedRunningTime="2026-07-12 19:29:51.007167241 +0000 UTC m=+599.600441340" Jul 12 19:29:51.025982 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:51.025913 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" podStartSLOduration=1.057683426 podStartE2EDuration="6.025891761s" totalImagesPullingTime="4.968208335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:29:51.023492084 +0000 UTC m=+599.616766194" watchObservedRunningTime="2026-07-12 19:29:51.025891761 +0000 UTC m=+599.619165861" Jul 12 19:29:51.041200 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:51.041135 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-zkrk2" podStartSLOduration=1.363967511 podStartE2EDuration="6.041112964s" totalImagesPullingTime="4.677145453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:29:51.038999154 +0000 UTC m=+599.632273251" watchObservedRunningTime="2026-07-12 19:29:51.041112964 +0000 UTC m=+599.634387063" Jul 12 19:29:51.988439 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:51.988322 2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-z58j6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 12 19:29:51.988439 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:51.988380 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" podUID="958bd3cb-cfc7-430c-b79d-1542b6d34e24" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 12 19:29:52.212749 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:52.212707 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:52.306854 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:52.306771 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:52.921120 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:52.921086 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt"] Jul 12 19:29:52.924203 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:52.924183 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:52.926833 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:52.926811 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-bnp5h\"" Jul 12 19:29:52.935569 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:52.935547 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt"] Jul 12 19:29:52.959083 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:52.959005 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:53.011316 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.011287 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916b483e-49cb-4db0-91d8-1c4d9b4020cd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.011675 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.011321 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq4sx\" (UniqueName: \"kubernetes.io/projected/916b483e-49cb-4db0-91d8-1c4d9b4020cd-kube-api-access-sq4sx\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.011675 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.011353 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916b483e-49cb-4db0-91d8-1c4d9b4020cd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.011675 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.011420 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916b483e-49cb-4db0-91d8-1c4d9b4020cd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.011675 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.011453 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916b483e-49cb-4db0-91d8-1c4d9b4020cd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.011675 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.011476 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916b483e-49cb-4db0-91d8-1c4d9b4020cd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.011675 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.011496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916b483e-49cb-4db0-91d8-1c4d9b4020cd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.011675 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.011550 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916b483e-49cb-4db0-91d8-1c4d9b4020cd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.011675 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.011595 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916b483e-49cb-4db0-91d8-1c4d9b4020cd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.112769 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.112736 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916b483e-49cb-4db0-91d8-1c4d9b4020cd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.112938 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.112794 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916b483e-49cb-4db0-91d8-1c4d9b4020cd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.112938 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.112820 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916b483e-49cb-4db0-91d8-1c4d9b4020cd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.112938 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.112846 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916b483e-49cb-4db0-91d8-1c4d9b4020cd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.112938 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.112874 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916b483e-49cb-4db0-91d8-1c4d9b4020cd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.112938 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.112900 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916b483e-49cb-4db0-91d8-1c4d9b4020cd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.113261 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.113231 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916b483e-49cb-4db0-91d8-1c4d9b4020cd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.113336 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.113260 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916b483e-49cb-4db0-91d8-1c4d9b4020cd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.113404 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.113364 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916b483e-49cb-4db0-91d8-1c4d9b4020cd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.113457 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.113444 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916b483e-49cb-4db0-91d8-1c4d9b4020cd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.113496 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.113472 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq4sx\" (UniqueName: \"kubernetes.io/projected/916b483e-49cb-4db0-91d8-1c4d9b4020cd-kube-api-access-sq4sx\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.113580 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.113560 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916b483e-49cb-4db0-91d8-1c4d9b4020cd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.113782 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.113761 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916b483e-49cb-4db0-91d8-1c4d9b4020cd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.114308 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.114284 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916b483e-49cb-4db0-91d8-1c4d9b4020cd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.115480 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.115449 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916b483e-49cb-4db0-91d8-1c4d9b4020cd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.116688 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.116663 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916b483e-49cb-4db0-91d8-1c4d9b4020cd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.121581 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.121552 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916b483e-49cb-4db0-91d8-1c4d9b4020cd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.122074 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.122053 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq4sx\" (UniqueName: \"kubernetes.io/projected/916b483e-49cb-4db0-91d8-1c4d9b4020cd-kube-api-access-sq4sx\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6vqt\" (UID: \"916b483e-49cb-4db0-91d8-1c4d9b4020cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.230522 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.230426 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:53.235432 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.235409 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:53.348203 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.348109 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:53.373647 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.373622 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt"] Jul 12 19:29:53.375780 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:29:53.375753 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916b483e_49cb_4db0_91d8_1c4d9b4020cd.slice/crio-3f3bda7dab32894ce441fb0cc42fbf6abd483c62d02de36b17795d44c0c1f54a WatchSource:0}: Error finding container 3f3bda7dab32894ce441fb0cc42fbf6abd483c62d02de36b17795d44c0c1f54a: Status 404 returned error can't find the container with id 3f3bda7dab32894ce441fb0cc42fbf6abd483c62d02de36b17795d44c0c1f54a Jul 12 19:29:53.997100 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:53.997064 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" event={"ID":"916b483e-49cb-4db0-91d8-1c4d9b4020cd","Type":"ContainerStarted","Data":"3f3bda7dab32894ce441fb0cc42fbf6abd483c62d02de36b17795d44c0c1f54a"} Jul 12 19:29:54.987787 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:54.987755 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:55.230936 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.230894 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:55.301171 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.301095 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ghffv"] Jul 12 19:29:55.307897 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.307875 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ghffv" Jul 12 19:29:55.312101 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.312078 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-lvhss\"" Jul 12 19:29:55.313419 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.313401 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 12 19:29:55.331079 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.331062 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-956xx"] Jul 12 19:29:55.334289 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.334272 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-956xx" Jul 12 19:29:55.338673 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.338654 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 12 19:29:55.338860 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.338840 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-6m5bn\"" Jul 12 19:29:55.343367 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.343342 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ghffv"] Jul 12 19:29:55.359672 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.359649 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-956xx"] Jul 12 19:29:55.360338 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.360312 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:55.435436 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.435398 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8df0c6a3-1593-4291-9e35-52e4b6a85a53-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-956xx\" (UID: \"8df0c6a3-1593-4291-9e35-52e4b6a85a53\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-956xx" Jul 12 19:29:55.435631 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.435501 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b8b400e-ccf0-4a14-b275-fa3bed8078ef-cert\") pod \"kserve-controller-manager-d6f4b65b8-ghffv\" (UID: \"8b8b400e-ccf0-4a14-b275-fa3bed8078ef\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ghffv" Jul 12 19:29:55.435631 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.435553 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hbqw\" (UniqueName: \"kubernetes.io/projected/8df0c6a3-1593-4291-9e35-52e4b6a85a53-kube-api-access-2hbqw\") pod \"llmisvc-controller-manager-777c6bd7d5-956xx\" (UID: \"8df0c6a3-1593-4291-9e35-52e4b6a85a53\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-956xx" Jul 12 19:29:55.435631 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.435600 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mfx5\" (UniqueName: \"kubernetes.io/projected/8b8b400e-ccf0-4a14-b275-fa3bed8078ef-kube-api-access-7mfx5\") pod \"kserve-controller-manager-d6f4b65b8-ghffv\" (UID: \"8b8b400e-ccf0-4a14-b275-fa3bed8078ef\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ghffv" Jul 12 19:29:55.536365 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.536313 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hbqw\" (UniqueName: \"kubernetes.io/projected/8df0c6a3-1593-4291-9e35-52e4b6a85a53-kube-api-access-2hbqw\") pod \"llmisvc-controller-manager-777c6bd7d5-956xx\" (UID: \"8df0c6a3-1593-4291-9e35-52e4b6a85a53\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-956xx" Jul 12 19:29:55.536365 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.536366 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mfx5\" (UniqueName: \"kubernetes.io/projected/8b8b400e-ccf0-4a14-b275-fa3bed8078ef-kube-api-access-7mfx5\") pod \"kserve-controller-manager-d6f4b65b8-ghffv\" (UID: \"8b8b400e-ccf0-4a14-b275-fa3bed8078ef\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ghffv" Jul 12 19:29:55.536599 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.536416 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8df0c6a3-1593-4291-9e35-52e4b6a85a53-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-956xx\" (UID: \"8df0c6a3-1593-4291-9e35-52e4b6a85a53\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-956xx" Jul 12 19:29:55.536599 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.536467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b8b400e-ccf0-4a14-b275-fa3bed8078ef-cert\") pod \"kserve-controller-manager-d6f4b65b8-ghffv\" (UID: \"8b8b400e-ccf0-4a14-b275-fa3bed8078ef\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ghffv" Jul 12 19:29:55.536599 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:29:55.536573 2564 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 12 19:29:55.536750 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:29:55.536642 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8b400e-ccf0-4a14-b275-fa3bed8078ef-cert podName:8b8b400e-ccf0-4a14-b275-fa3bed8078ef nodeName:}" failed. No retries permitted until 2026-07-12 19:29:56.03662238 +0000 UTC m=+604.629896456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8b8b400e-ccf0-4a14-b275-fa3bed8078ef-cert") pod "kserve-controller-manager-d6f4b65b8-ghffv" (UID: "8b8b400e-ccf0-4a14-b275-fa3bed8078ef") : secret "kserve-webhook-server-cert" not found Jul 12 19:29:55.539210 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.539187 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8df0c6a3-1593-4291-9e35-52e4b6a85a53-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-956xx\" (UID: \"8df0c6a3-1593-4291-9e35-52e4b6a85a53\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-956xx" Jul 12 19:29:55.551467 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.551405 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mfx5\" (UniqueName: \"kubernetes.io/projected/8b8b400e-ccf0-4a14-b275-fa3bed8078ef-kube-api-access-7mfx5\") pod \"kserve-controller-manager-d6f4b65b8-ghffv\" (UID: \"8b8b400e-ccf0-4a14-b275-fa3bed8078ef\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ghffv" Jul 12 19:29:55.551821 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.551797 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hbqw\" (UniqueName: \"kubernetes.io/projected/8df0c6a3-1593-4291-9e35-52e4b6a85a53-kube-api-access-2hbqw\") pod \"llmisvc-controller-manager-777c6bd7d5-956xx\" (UID: \"8df0c6a3-1593-4291-9e35-52e4b6a85a53\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-956xx" Jul 12 19:29:55.645529 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.645496 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-956xx" Jul 12 19:29:55.713301 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.713263 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:29:55.713379 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.713363 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:29:55.713420 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.713411 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:29:55.775741 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:55.775718 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-956xx"] Jul 12 19:29:55.777583 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:29:55.777560 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8df0c6a3_1593_4291_9e35_52e4b6a85a53.slice/crio-a9e11d08fe9b3380b8eaf48eb75b9087f43d17ddfbedfcbdf305e5dd009b73fa WatchSource:0}: Error finding container a9e11d08fe9b3380b8eaf48eb75b9087f43d17ddfbedfcbdf305e5dd009b73fa: Status 404 returned error can't find the container with id a9e11d08fe9b3380b8eaf48eb75b9087f43d17ddfbedfcbdf305e5dd009b73fa Jul 12 19:29:56.007282 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:56.007245 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-956xx" event={"ID":"8df0c6a3-1593-4291-9e35-52e4b6a85a53","Type":"ContainerStarted","Data":"a9e11d08fe9b3380b8eaf48eb75b9087f43d17ddfbedfcbdf305e5dd009b73fa"} Jul 12 19:29:56.008913 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:56.008883 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" event={"ID":"916b483e-49cb-4db0-91d8-1c4d9b4020cd","Type":"ContainerStarted","Data":"c2bfb6c9e17334fbb577b57c7fd5e681f83fcc559249c0e26c245db5b5b088e4"} Jul 12 19:29:56.039705 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:56.039677 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b8b400e-ccf0-4a14-b275-fa3bed8078ef-cert\") pod \"kserve-controller-manager-d6f4b65b8-ghffv\" (UID: \"8b8b400e-ccf0-4a14-b275-fa3bed8078ef\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ghffv" Jul 12 19:29:56.042498 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:56.042452 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" podStartSLOduration=1.707161534 podStartE2EDuration="4.042435806s" totalImagesPullingTime="2.335274272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:29:56.039725441 +0000 UTC m=+604.632999541" watchObservedRunningTime="2026-07-12 19:29:56.042435806 +0000 UTC m=+604.635709904" Jul 12 19:29:56.042725 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:56.042703 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b8b400e-ccf0-4a14-b275-fa3bed8078ef-cert\") pod \"kserve-controller-manager-d6f4b65b8-ghffv\" (UID: \"8b8b400e-ccf0-4a14-b275-fa3bed8078ef\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ghffv" Jul 12 19:29:56.220250 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:56.220217 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ghffv" Jul 12 19:29:56.236102 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:56.236070 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:56.240796 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:56.240771 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:56.374136 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:56.373988 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ghffv"] Jul 12 19:29:56.376836 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:29:56.376810 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8b400e_ccf0_4a14_b275_fa3bed8078ef.slice/crio-cd305ce2ae9283ce40179d51c597ee573fbeceba2473eaac1f21c29545acb5fe WatchSource:0}: Error finding container cd305ce2ae9283ce40179d51c597ee573fbeceba2473eaac1f21c29545acb5fe: Status 404 returned error can't find the container with id cd305ce2ae9283ce40179d51c597ee573fbeceba2473eaac1f21c29545acb5fe Jul 12 19:29:56.573347 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:56.573268 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:56.670629 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:56.670574 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:56.771670 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:56.771626 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:57.016954 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.016918 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ghffv" event={"ID":"8b8b400e-ccf0-4a14-b275-fa3bed8078ef","Type":"ContainerStarted","Data":"cd305ce2ae9283ce40179d51c597ee573fbeceba2473eaac1f21c29545acb5fe"} Jul 12 19:29:57.017406 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.017245 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:57.018262 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.018244 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6vqt" Jul 12 19:29:57.330849 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.330755 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-z58j6"] Jul 12 19:29:57.331054 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.330996 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" podUID="958bd3cb-cfc7-430c-b79d-1542b6d34e24" containerName="discovery" containerID="cri-o://014cf2f3c02a6956856878af6249c1c0dc8ccb1bc76e989cccc8d31aeb917269" gracePeriod=30 Jul 12 19:29:57.606519 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.606488 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:57.756810 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.756774 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/958bd3cb-cfc7-430c-b79d-1542b6d34e24-local-certs\") pod \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " Jul 12 19:29:57.756810 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.756812 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/958bd3cb-cfc7-430c-b79d-1542b6d34e24-cacerts\") pod \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " Jul 12 19:29:57.757070 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.756882 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-kubeconfig\") pod \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " Jul 12 19:29:57.757070 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.756914 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-token\") pod \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " Jul 12 19:29:57.757070 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.756939 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ff94t\" (UniqueName: \"kubernetes.io/projected/958bd3cb-cfc7-430c-b79d-1542b6d34e24-kube-api-access-ff94t\") pod \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " Jul 12 19:29:57.757228 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.757081 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-csr-ca-configmap\") pod \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " Jul 12 19:29:57.757228 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.757144 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-csr-dns-cert\") pod \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\" (UID: \"958bd3cb-cfc7-430c-b79d-1542b6d34e24\") " Jul 12 19:29:57.757603 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.757550 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "958bd3cb-cfc7-430c-b79d-1542b6d34e24" (UID: "958bd3cb-cfc7-430c-b79d-1542b6d34e24"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:29:57.759562 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.759529 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958bd3cb-cfc7-430c-b79d-1542b6d34e24-local-certs" (OuterVolumeSpecName: "local-certs") pod "958bd3cb-cfc7-430c-b79d-1542b6d34e24" (UID: "958bd3cb-cfc7-430c-b79d-1542b6d34e24"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:29:57.759649 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.759596 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/958bd3cb-cfc7-430c-b79d-1542b6d34e24-kube-api-access-ff94t" (OuterVolumeSpecName: "kube-api-access-ff94t") pod "958bd3cb-cfc7-430c-b79d-1542b6d34e24" (UID: "958bd3cb-cfc7-430c-b79d-1542b6d34e24"). InnerVolumeSpecName "kube-api-access-ff94t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:29:57.759856 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.759825 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-token" (OuterVolumeSpecName: "istio-token") pod "958bd3cb-cfc7-430c-b79d-1542b6d34e24" (UID: "958bd3cb-cfc7-430c-b79d-1542b6d34e24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:29:57.759939 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.759894 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "958bd3cb-cfc7-430c-b79d-1542b6d34e24" (UID: "958bd3cb-cfc7-430c-b79d-1542b6d34e24"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:29:57.759992 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.759980 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "958bd3cb-cfc7-430c-b79d-1542b6d34e24" (UID: "958bd3cb-cfc7-430c-b79d-1542b6d34e24"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:29:57.760081 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.760040 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/958bd3cb-cfc7-430c-b79d-1542b6d34e24-cacerts" (OuterVolumeSpecName: "cacerts") pod "958bd3cb-cfc7-430c-b79d-1542b6d34e24" (UID: "958bd3cb-cfc7-430c-b79d-1542b6d34e24"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:29:57.858779 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.858689 2564 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-kubeconfig\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:29:57.858779 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.858724 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-token\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:29:57.858779 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.858733 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff94t\" (UniqueName: \"kubernetes.io/projected/958bd3cb-cfc7-430c-b79d-1542b6d34e24-kube-api-access-ff94t\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:29:57.858779 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.858742 2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-csr-ca-configmap\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:29:57.858779 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.858756 2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/958bd3cb-cfc7-430c-b79d-1542b6d34e24-istio-csr-dns-cert\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:29:57.858779 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.858769 2564 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/958bd3cb-cfc7-430c-b79d-1542b6d34e24-local-certs\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:29:57.858779 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:57.858781 2564 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/958bd3cb-cfc7-430c-b79d-1542b6d34e24-cacerts\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:29:58.025030 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:58.024997 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" Jul 12 19:29:58.025484 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:58.025039 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" event={"ID":"958bd3cb-cfc7-430c-b79d-1542b6d34e24","Type":"ContainerDied","Data":"014cf2f3c02a6956856878af6249c1c0dc8ccb1bc76e989cccc8d31aeb917269"} Jul 12 19:29:58.025484 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:58.025103 2564 scope.go:117] "RemoveContainer" containerID="014cf2f3c02a6956856878af6249c1c0dc8ccb1bc76e989cccc8d31aeb917269" Jul 12 19:29:58.025484 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:58.024994 2564 generic.go:356] "Generic (PLEG): container finished" podID="958bd3cb-cfc7-430c-b79d-1542b6d34e24" containerID="014cf2f3c02a6956856878af6249c1c0dc8ccb1bc76e989cccc8d31aeb917269" exitCode=0 Jul 12 19:29:58.025484 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:58.025378 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z58j6" event={"ID":"958bd3cb-cfc7-430c-b79d-1542b6d34e24","Type":"ContainerDied","Data":"8adeb08c8f1e7e35383235b8ef90956792fefe8da547d7716754038a61a36d7b"} Jul 12 19:29:58.037968 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:58.037947 2564 scope.go:117] "RemoveContainer" containerID="014cf2f3c02a6956856878af6249c1c0dc8ccb1bc76e989cccc8d31aeb917269" Jul 12 19:29:58.038320 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:29:58.038289 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"014cf2f3c02a6956856878af6249c1c0dc8ccb1bc76e989cccc8d31aeb917269\": container with ID starting with 014cf2f3c02a6956856878af6249c1c0dc8ccb1bc76e989cccc8d31aeb917269 not found: ID does not exist" containerID="014cf2f3c02a6956856878af6249c1c0dc8ccb1bc76e989cccc8d31aeb917269" Jul 12 19:29:58.038421 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:58.038321 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014cf2f3c02a6956856878af6249c1c0dc8ccb1bc76e989cccc8d31aeb917269"} err="failed to get container status \"014cf2f3c02a6956856878af6249c1c0dc8ccb1bc76e989cccc8d31aeb917269\": rpc error: code = NotFound desc = could not find container \"014cf2f3c02a6956856878af6249c1c0dc8ccb1bc76e989cccc8d31aeb917269\": container with ID starting with 014cf2f3c02a6956856878af6249c1c0dc8ccb1bc76e989cccc8d31aeb917269 not found: ID does not exist" Jul 12 19:29:58.043333 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:58.043308 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-z58j6"] Jul 12 19:29:58.047282 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:58.047257 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-z58j6"] Jul 12 19:29:59.915756 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:29:59.915725 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="958bd3cb-cfc7-430c-b79d-1542b6d34e24" path="/var/lib/kubelet/pods/958bd3cb-cfc7-430c-b79d-1542b6d34e24/volumes" Jul 12 19:30:00.037246 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:00.037210 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ghffv" event={"ID":"8b8b400e-ccf0-4a14-b275-fa3bed8078ef","Type":"ContainerStarted","Data":"bdf4bed3afdef7c3292f1414b698a18dd8117fd9f7b13928c5cc81da76097567"} Jul 12 19:30:00.037442 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:00.037297 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ghffv" Jul 12 19:30:00.038770 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:00.038746 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-956xx" event={"ID":"8df0c6a3-1593-4291-9e35-52e4b6a85a53","Type":"ContainerStarted","Data":"601d40ad45b448584c9add325db325b665a2773cb78654cc35bd49a33f4a4884"} Jul 12 19:30:00.038868 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:00.038792 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-956xx" Jul 12 19:30:00.054587 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:00.054535 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ghffv" podStartSLOduration=2.187327367 podStartE2EDuration="5.054519387s" totalImagesPullingTime="2.86719202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:30:00.053123108 +0000 UTC m=+608.646397206" watchObservedRunningTime="2026-07-12 19:30:00.054519387 +0000 UTC m=+608.647793484" Jul 12 19:30:00.071441 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:00.071389 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-956xx" podStartSLOduration=1.609265328 podStartE2EDuration="5.071371861s" totalImagesPullingTime="3.462106533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:30:00.069979894 +0000 UTC m=+608.663253993" watchObservedRunningTime="2026-07-12 19:30:00.071371861 +0000 UTC m=+608.664645959" Jul 12 19:30:01.989028 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:01.988998 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-bm6ks" Jul 12 19:30:01.990809 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:01.990780 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-zkrk2" Jul 12 19:30:03.711896 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:03.711861 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:30:03.762264 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:03.762231 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:30:05.433156 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:05.433123 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:30:05.534194 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:05.534162 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:30:31.044850 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:31.044819 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-956xx" Jul 12 19:30:31.048191 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:31.048170 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ghffv" Jul 12 19:30:45.078945 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.078895 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:30:45.177426 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.177388 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:30:45.207567 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.207535 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b95mw"] Jul 12 19:30:45.207854 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.207842 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="958bd3cb-cfc7-430c-b79d-1542b6d34e24" containerName="discovery" Jul 12 19:30:45.207907 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.207855 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="958bd3cb-cfc7-430c-b79d-1542b6d34e24" containerName="discovery" Jul 12 19:30:45.207907 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.207900 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="958bd3cb-cfc7-430c-b79d-1542b6d34e24" containerName="discovery" Jul 12 19:30:45.209792 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.209769 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-b95mw" Jul 12 19:30:45.212545 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.212525 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 12 19:30:45.213766 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.213731 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-pp4hh\"" Jul 12 19:30:45.213977 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.213961 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 12 19:30:45.219191 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.219159 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b95mw"] Jul 12 19:30:45.248347 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.248317 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:30:45.298639 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.298614 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzvtd\" (UniqueName: \"kubernetes.io/projected/7cc070a4-1aac-4587-9773-c02d04cbd11f-kube-api-access-qzvtd\") pod \"kuadrant-operator-catalog-b95mw\" (UID: \"7cc070a4-1aac-4587-9773-c02d04cbd11f\") " pod="kuadrant-system/kuadrant-operator-catalog-b95mw" Jul 12 19:30:45.399650 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.399539 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzvtd\" (UniqueName: \"kubernetes.io/projected/7cc070a4-1aac-4587-9773-c02d04cbd11f-kube-api-access-qzvtd\") pod \"kuadrant-operator-catalog-b95mw\" (UID: \"7cc070a4-1aac-4587-9773-c02d04cbd11f\") " pod="kuadrant-system/kuadrant-operator-catalog-b95mw" Jul 12 19:30:45.407537 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.407514 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzvtd\" (UniqueName: \"kubernetes.io/projected/7cc070a4-1aac-4587-9773-c02d04cbd11f-kube-api-access-qzvtd\") pod \"kuadrant-operator-catalog-b95mw\" (UID: \"7cc070a4-1aac-4587-9773-c02d04cbd11f\") " pod="kuadrant-system/kuadrant-operator-catalog-b95mw" Jul 12 19:30:45.521327 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.521292 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-b95mw" Jul 12 19:30:45.649397 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.649374 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b95mw"] Jul 12 19:30:45.651826 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:30:45.651773 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc070a4_1aac_4587_9773_c02d04cbd11f.slice/crio-a3096268fccb0fde8f23344b4c3b5bea730f80fbb6cfbe54b7d31357026329c0 WatchSource:0}: Error finding container a3096268fccb0fde8f23344b4c3b5bea730f80fbb6cfbe54b7d31357026329c0: Status 404 returned error can't find the container with id a3096268fccb0fde8f23344b4c3b5bea730f80fbb6cfbe54b7d31357026329c0 Jul 12 19:30:45.733093 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.733053 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b95mw"] Jul 12 19:30:45.776223 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.776190 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-f9lgb"] Jul 12 19:30:45.779484 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.779466 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-f9lgb" Jul 12 19:30:45.790673 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.790637 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-f9lgb"] Jul 12 19:30:45.903300 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:45.903278 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgm8q\" (UniqueName: \"kubernetes.io/projected/6e13e221-b298-49fa-9392-f3ce487b978b-kube-api-access-kgm8q\") pod \"kuadrant-operator-catalog-f9lgb\" (UID: \"6e13e221-b298-49fa-9392-f3ce487b978b\") " pod="kuadrant-system/kuadrant-operator-catalog-f9lgb" Jul 12 19:30:46.004149 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:46.004117 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgm8q\" (UniqueName: \"kubernetes.io/projected/6e13e221-b298-49fa-9392-f3ce487b978b-kube-api-access-kgm8q\") pod \"kuadrant-operator-catalog-f9lgb\" (UID: \"6e13e221-b298-49fa-9392-f3ce487b978b\") " pod="kuadrant-system/kuadrant-operator-catalog-f9lgb" Jul 12 19:30:46.011765 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:46.011743 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgm8q\" (UniqueName: \"kubernetes.io/projected/6e13e221-b298-49fa-9392-f3ce487b978b-kube-api-access-kgm8q\") pod \"kuadrant-operator-catalog-f9lgb\" (UID: \"6e13e221-b298-49fa-9392-f3ce487b978b\") " pod="kuadrant-system/kuadrant-operator-catalog-f9lgb" Jul 12 19:30:46.088547 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:46.088519 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-f9lgb" Jul 12 19:30:46.217788 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:46.217764 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-f9lgb"] Jul 12 19:30:46.231944 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:46.231915 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-b95mw" event={"ID":"7cc070a4-1aac-4587-9773-c02d04cbd11f","Type":"ContainerStarted","Data":"a3096268fccb0fde8f23344b4c3b5bea730f80fbb6cfbe54b7d31357026329c0"} Jul 12 19:30:46.251689 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:30:46.251657 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e13e221_b298_49fa_9392_f3ce487b978b.slice/crio-a606c62cd845108e87f3323b17693a76e3a06be017698b4d3020d462f040c650 WatchSource:0}: Error finding container a606c62cd845108e87f3323b17693a76e3a06be017698b4d3020d462f040c650: Status 404 returned error can't find the container with id a606c62cd845108e87f3323b17693a76e3a06be017698b4d3020d462f040c650 Jul 12 19:30:47.239093 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:47.239053 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-f9lgb" event={"ID":"6e13e221-b298-49fa-9392-f3ce487b978b","Type":"ContainerStarted","Data":"a606c62cd845108e87f3323b17693a76e3a06be017698b4d3020d462f040c650"} Jul 12 19:30:48.244167 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:48.244131 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-b95mw" event={"ID":"7cc070a4-1aac-4587-9773-c02d04cbd11f","Type":"ContainerStarted","Data":"71df201009345f85bb3a0422ddad3395d78eea05d5b06e32be9bd6fcf4f45331"} Jul 12 19:30:48.244667 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:48.244194 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-b95mw" podUID="7cc070a4-1aac-4587-9773-c02d04cbd11f" containerName="registry-server" containerID="cri-o://71df201009345f85bb3a0422ddad3395d78eea05d5b06e32be9bd6fcf4f45331" gracePeriod=2 Jul 12 19:30:48.245787 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:48.245754 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-f9lgb" event={"ID":"6e13e221-b298-49fa-9392-f3ce487b978b","Type":"ContainerStarted","Data":"89d77164890ee0bd2b6a1c3be87a51ed6e505d91bc221f5680f5ac6e9e332278"} Jul 12 19:30:48.259706 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:48.259659 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-b95mw" podStartSLOduration=1.335253186 podStartE2EDuration="3.259643578s" totalImagesPullingTime="1.924390392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:30:48.258096976 +0000 UTC m=+656.851371075" watchObservedRunningTime="2026-07-12 19:30:48.259643578 +0000 UTC m=+656.852917675" Jul 12 19:30:48.273253 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:48.273198 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-f9lgb" podStartSLOduration=1.9457630620000002 podStartE2EDuration="3.273179814s" totalImagesPullingTime="1.327416752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:30:48.272557246 +0000 UTC m=+656.865831346" watchObservedRunningTime="2026-07-12 19:30:48.273179814 +0000 UTC m=+656.866453913" Jul 12 19:30:48.479277 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:48.479253 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-b95mw" Jul 12 19:30:48.531588 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:48.531496 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzvtd\" (UniqueName: \"kubernetes.io/projected/7cc070a4-1aac-4587-9773-c02d04cbd11f-kube-api-access-qzvtd\") pod \"7cc070a4-1aac-4587-9773-c02d04cbd11f\" (UID: \"7cc070a4-1aac-4587-9773-c02d04cbd11f\") " Jul 12 19:30:48.533739 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:48.533715 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc070a4-1aac-4587-9773-c02d04cbd11f-kube-api-access-qzvtd" (OuterVolumeSpecName: "kube-api-access-qzvtd") pod "7cc070a4-1aac-4587-9773-c02d04cbd11f" (UID: "7cc070a4-1aac-4587-9773-c02d04cbd11f"). InnerVolumeSpecName "kube-api-access-qzvtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:30:48.632899 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:48.632862 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzvtd\" (UniqueName: \"kubernetes.io/projected/7cc070a4-1aac-4587-9773-c02d04cbd11f-kube-api-access-qzvtd\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:30:49.251398 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:49.251363 2564 generic.go:356] "Generic (PLEG): container finished" podID="7cc070a4-1aac-4587-9773-c02d04cbd11f" containerID="71df201009345f85bb3a0422ddad3395d78eea05d5b06e32be9bd6fcf4f45331" exitCode=0 Jul 12 19:30:49.251805 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:49.251425 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-b95mw" Jul 12 19:30:49.251805 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:49.251455 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-b95mw" event={"ID":"7cc070a4-1aac-4587-9773-c02d04cbd11f","Type":"ContainerDied","Data":"71df201009345f85bb3a0422ddad3395d78eea05d5b06e32be9bd6fcf4f45331"} Jul 12 19:30:49.251805 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:49.251503 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-b95mw" event={"ID":"7cc070a4-1aac-4587-9773-c02d04cbd11f","Type":"ContainerDied","Data":"a3096268fccb0fde8f23344b4c3b5bea730f80fbb6cfbe54b7d31357026329c0"} Jul 12 19:30:49.251805 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:49.251523 2564 scope.go:117] "RemoveContainer" containerID="71df201009345f85bb3a0422ddad3395d78eea05d5b06e32be9bd6fcf4f45331" Jul 12 19:30:49.260365 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:49.260346 2564 scope.go:117] "RemoveContainer" containerID="71df201009345f85bb3a0422ddad3395d78eea05d5b06e32be9bd6fcf4f45331" Jul 12 19:30:49.260605 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:30:49.260586 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71df201009345f85bb3a0422ddad3395d78eea05d5b06e32be9bd6fcf4f45331\": container with ID starting with 71df201009345f85bb3a0422ddad3395d78eea05d5b06e32be9bd6fcf4f45331 not found: ID does not exist" containerID="71df201009345f85bb3a0422ddad3395d78eea05d5b06e32be9bd6fcf4f45331" Jul 12 19:30:49.260663 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:49.260618 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71df201009345f85bb3a0422ddad3395d78eea05d5b06e32be9bd6fcf4f45331"} err="failed to get container status \"71df201009345f85bb3a0422ddad3395d78eea05d5b06e32be9bd6fcf4f45331\": rpc error: code = NotFound desc = could not find container \"71df201009345f85bb3a0422ddad3395d78eea05d5b06e32be9bd6fcf4f45331\": container with ID starting with 71df201009345f85bb3a0422ddad3395d78eea05d5b06e32be9bd6fcf4f45331 not found: ID does not exist" Jul 12 19:30:49.270802 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:49.270781 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b95mw"] Jul 12 19:30:49.274568 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:49.274550 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b95mw"] Jul 12 19:30:49.915907 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:49.915870 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cc070a4-1aac-4587-9773-c02d04cbd11f" path="/var/lib/kubelet/pods/7cc070a4-1aac-4587-9773-c02d04cbd11f/volumes" Jul 12 19:30:56.089491 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:56.089441 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-f9lgb" Jul 12 19:30:56.089943 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:56.089517 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-f9lgb" Jul 12 19:30:56.111926 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:56.111904 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-f9lgb" Jul 12 19:30:56.302159 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:30:56.302133 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-f9lgb" Jul 12 19:31:00.748699 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:00.748666 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb"] Jul 12 19:31:00.749101 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:00.748939 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc070a4-1aac-4587-9773-c02d04cbd11f" containerName="registry-server" Jul 12 19:31:00.749101 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:00.748950 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc070a4-1aac-4587-9773-c02d04cbd11f" containerName="registry-server" Jul 12 19:31:00.749101 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:00.749005 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cc070a4-1aac-4587-9773-c02d04cbd11f" containerName="registry-server" Jul 12 19:31:00.752345 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:00.752329 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb" Jul 12 19:31:00.754778 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:00.754751 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qlwpj\"" Jul 12 19:31:00.759424 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:00.759385 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb"] Jul 12 19:31:00.824386 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:00.824354 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e8eb60c-253a-4201-b30c-561a435b44ab-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb\" (UID: \"3e8eb60c-253a-4201-b30c-561a435b44ab\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb" Jul 12 19:31:00.824386 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:00.824396 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xm9l\" (UniqueName: \"kubernetes.io/projected/3e8eb60c-253a-4201-b30c-561a435b44ab-kube-api-access-8xm9l\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb\" (UID: \"3e8eb60c-253a-4201-b30c-561a435b44ab\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb" Jul 12 19:31:00.824592 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:00.824422 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e8eb60c-253a-4201-b30c-561a435b44ab-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb\" (UID: \"3e8eb60c-253a-4201-b30c-561a435b44ab\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb" Jul 12 19:31:00.925030 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:00.924997 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e8eb60c-253a-4201-b30c-561a435b44ab-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb\" (UID: \"3e8eb60c-253a-4201-b30c-561a435b44ab\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb" Jul 12 19:31:00.925189 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:00.925065 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xm9l\" (UniqueName: \"kubernetes.io/projected/3e8eb60c-253a-4201-b30c-561a435b44ab-kube-api-access-8xm9l\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb\" (UID: \"3e8eb60c-253a-4201-b30c-561a435b44ab\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb" Jul 12 19:31:00.925189 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:00.925089 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e8eb60c-253a-4201-b30c-561a435b44ab-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb\" (UID: \"3e8eb60c-253a-4201-b30c-561a435b44ab\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb" Jul 12 19:31:00.925373 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:00.925355 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e8eb60c-253a-4201-b30c-561a435b44ab-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb\" (UID: \"3e8eb60c-253a-4201-b30c-561a435b44ab\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb" Jul 12 19:31:00.925406 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:00.925388 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e8eb60c-253a-4201-b30c-561a435b44ab-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb\" (UID: \"3e8eb60c-253a-4201-b30c-561a435b44ab\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb" Jul 12 19:31:00.933499 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:00.933467 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xm9l\" (UniqueName: \"kubernetes.io/projected/3e8eb60c-253a-4201-b30c-561a435b44ab-kube-api-access-8xm9l\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb\" (UID: \"3e8eb60c-253a-4201-b30c-561a435b44ab\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb" Jul 12 19:31:01.062170 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.062090 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb" Jul 12 19:31:01.189669 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.189531 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb"] Jul 12 19:31:01.192238 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:31:01.192087 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8eb60c_253a_4201_b30c_561a435b44ab.slice/crio-52a1355aa5fcd6618b00876befce6f9b536c11adaf96ac49aff20fcee17e2862 WatchSource:0}: Error finding container 52a1355aa5fcd6618b00876befce6f9b536c11adaf96ac49aff20fcee17e2862: Status 404 returned error can't find the container with id 52a1355aa5fcd6618b00876befce6f9b536c11adaf96ac49aff20fcee17e2862 Jul 12 19:31:01.300973 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.300948 2564 generic.go:356] "Generic (PLEG): container finished" podID="3e8eb60c-253a-4201-b30c-561a435b44ab" containerID="8ec7c4412698e784e1972efc9a4cf3f91b23893c04232d2263f9543268eec8a1" exitCode=0 Jul 12 19:31:01.301085 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.301001 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb" event={"ID":"3e8eb60c-253a-4201-b30c-561a435b44ab","Type":"ContainerDied","Data":"8ec7c4412698e784e1972efc9a4cf3f91b23893c04232d2263f9543268eec8a1"} Jul 12 19:31:01.301183 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.301083 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb" event={"ID":"3e8eb60c-253a-4201-b30c-561a435b44ab","Type":"ContainerStarted","Data":"52a1355aa5fcd6618b00876befce6f9b536c11adaf96ac49aff20fcee17e2862"} Jul 12 19:31:01.356664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.356604 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl"] Jul 12 19:31:01.359206 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.359188 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl" Jul 12 19:31:01.367658 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.367625 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl"] Jul 12 19:31:01.430798 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.430768 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl\" (UID: \"a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl" Jul 12 19:31:01.430921 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.430803 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl\" (UID: \"a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl" Jul 12 19:31:01.430921 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.430829 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh95q\" (UniqueName: \"kubernetes.io/projected/a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67-kube-api-access-gh95q\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl\" (UID: \"a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl" Jul 12 19:31:01.532213 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.532183 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl\" (UID: \"a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl" Jul 12 19:31:01.532213 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.532212 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl\" (UID: \"a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl" Jul 12 19:31:01.532356 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.532230 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh95q\" (UniqueName: \"kubernetes.io/projected/a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67-kube-api-access-gh95q\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl\" (UID: \"a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl" Jul 12 19:31:01.532533 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.532515 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl\" (UID: \"a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl" Jul 12 19:31:01.532581 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.532562 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl\" (UID: \"a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl" Jul 12 19:31:01.539840 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.539822 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh95q\" (UniqueName: \"kubernetes.io/projected/a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67-kube-api-access-gh95q\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl\" (UID: \"a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl" Jul 12 19:31:01.668826 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.668802 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl" Jul 12 19:31:01.749613 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.749586 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc"] Jul 12 19:31:01.753457 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.753435 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc" Jul 12 19:31:01.760339 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.760313 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc"] Jul 12 19:31:01.796789 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.796766 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl"] Jul 12 19:31:01.799069 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:31:01.799041 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9fefec5_6d6d_4837_a7e4_f9fb93b6ce67.slice/crio-ec65b33129ce77c05f55c19436b1ef72d6ebdd147c3bda6a718a9307f7f90f4a WatchSource:0}: Error finding container ec65b33129ce77c05f55c19436b1ef72d6ebdd147c3bda6a718a9307f7f90f4a: Status 404 returned error can't find the container with id ec65b33129ce77c05f55c19436b1ef72d6ebdd147c3bda6a718a9307f7f90f4a Jul 12 19:31:01.834915 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.834874 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/673bd84f-85af-496f-b902-c74c065574fa-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc\" (UID: \"673bd84f-85af-496f-b902-c74c065574fa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc" Jul 12 19:31:01.834915 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.834917 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/673bd84f-85af-496f-b902-c74c065574fa-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc\" (UID: \"673bd84f-85af-496f-b902-c74c065574fa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc" Jul 12 19:31:01.835113 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.834953 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv74l\" (UniqueName: \"kubernetes.io/projected/673bd84f-85af-496f-b902-c74c065574fa-kube-api-access-jv74l\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc\" (UID: \"673bd84f-85af-496f-b902-c74c065574fa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc" Jul 12 19:31:01.936041 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.935945 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv74l\" (UniqueName: \"kubernetes.io/projected/673bd84f-85af-496f-b902-c74c065574fa-kube-api-access-jv74l\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc\" (UID: \"673bd84f-85af-496f-b902-c74c065574fa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc" Jul 12 19:31:01.936041 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.936038 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/673bd84f-85af-496f-b902-c74c065574fa-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc\" (UID: \"673bd84f-85af-496f-b902-c74c065574fa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc" Jul 12 19:31:01.936255 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.936059 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/673bd84f-85af-496f-b902-c74c065574fa-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc\" (UID: \"673bd84f-85af-496f-b902-c74c065574fa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc" Jul 12 19:31:01.936350 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.936334 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/673bd84f-85af-496f-b902-c74c065574fa-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc\" (UID: \"673bd84f-85af-496f-b902-c74c065574fa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc" Jul 12 19:31:01.936400 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.936365 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/673bd84f-85af-496f-b902-c74c065574fa-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc\" (UID: \"673bd84f-85af-496f-b902-c74c065574fa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc" Jul 12 19:31:01.943743 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:01.943727 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv74l\" (UniqueName: \"kubernetes.io/projected/673bd84f-85af-496f-b902-c74c065574fa-kube-api-access-jv74l\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc\" (UID: \"673bd84f-85af-496f-b902-c74c065574fa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc" Jul 12 19:31:02.065744 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.065721 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc" Jul 12 19:31:02.146934 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.146901 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs"] Jul 12 19:31:02.149490 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.149466 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs" Jul 12 19:31:02.158423 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.158399 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs"] Jul 12 19:31:02.198511 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.198485 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc"] Jul 12 19:31:02.200087 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:31:02.200061 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod673bd84f_85af_496f_b902_c74c065574fa.slice/crio-cea60a8bc09f96b352b064be554db1425715268bfe78a37e1a03785f47fe8ae3 WatchSource:0}: Error finding container cea60a8bc09f96b352b064be554db1425715268bfe78a37e1a03785f47fe8ae3: Status 404 returned error can't find the container with id cea60a8bc09f96b352b064be554db1425715268bfe78a37e1a03785f47fe8ae3 Jul 12 19:31:02.238153 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.238130 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1af07a3-642b-4014-8dc0-bdec0818d9b5-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs\" (UID: \"e1af07a3-642b-4014-8dc0-bdec0818d9b5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs" Jul 12 19:31:02.238254 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.238185 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1af07a3-642b-4014-8dc0-bdec0818d9b5-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs\" (UID: \"e1af07a3-642b-4014-8dc0-bdec0818d9b5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs" Jul 12 19:31:02.238254 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.238231 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsrz6\" (UniqueName: \"kubernetes.io/projected/e1af07a3-642b-4014-8dc0-bdec0818d9b5-kube-api-access-fsrz6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs\" (UID: \"e1af07a3-642b-4014-8dc0-bdec0818d9b5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs" Jul 12 19:31:02.308150 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.308121 2564 generic.go:356] "Generic (PLEG): container finished" podID="3e8eb60c-253a-4201-b30c-561a435b44ab" containerID="968bd7af9fd3e271597f968783b18b823e85edf5c026bf98d18447730d669ac0" exitCode=0 Jul 12 19:31:02.308268 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.308199 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb" event={"ID":"3e8eb60c-253a-4201-b30c-561a435b44ab","Type":"ContainerDied","Data":"968bd7af9fd3e271597f968783b18b823e85edf5c026bf98d18447730d669ac0"} Jul 12 19:31:02.309589 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.309567 2564 generic.go:356] "Generic (PLEG): container finished" podID="673bd84f-85af-496f-b902-c74c065574fa" containerID="68bdc7dbcf64ce6ca9aeaa4c23c4c04feb7dadf93480e9982e894e563cf6c7f7" exitCode=0 Jul 12 19:31:02.309718 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.309661 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc" event={"ID":"673bd84f-85af-496f-b902-c74c065574fa","Type":"ContainerDied","Data":"68bdc7dbcf64ce6ca9aeaa4c23c4c04feb7dadf93480e9982e894e563cf6c7f7"} Jul 12 19:31:02.309827 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.309744 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc" event={"ID":"673bd84f-85af-496f-b902-c74c065574fa","Type":"ContainerStarted","Data":"cea60a8bc09f96b352b064be554db1425715268bfe78a37e1a03785f47fe8ae3"} Jul 12 19:31:02.311332 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.311312 2564 generic.go:356] "Generic (PLEG): container finished" podID="a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67" containerID="1078e84864024d23376b2bde510b03af56fadd5e202aeb3221057eb556aea49a" exitCode=0 Jul 12 19:31:02.311408 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.311349 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl" event={"ID":"a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67","Type":"ContainerDied","Data":"1078e84864024d23376b2bde510b03af56fadd5e202aeb3221057eb556aea49a"} Jul 12 19:31:02.311408 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.311381 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl" event={"ID":"a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67","Type":"ContainerStarted","Data":"ec65b33129ce77c05f55c19436b1ef72d6ebdd147c3bda6a718a9307f7f90f4a"} Jul 12 19:31:02.339229 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.339199 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1af07a3-642b-4014-8dc0-bdec0818d9b5-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs\" (UID: \"e1af07a3-642b-4014-8dc0-bdec0818d9b5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs" Jul 12 19:31:02.339332 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.339249 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsrz6\" (UniqueName: \"kubernetes.io/projected/e1af07a3-642b-4014-8dc0-bdec0818d9b5-kube-api-access-fsrz6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs\" (UID: \"e1af07a3-642b-4014-8dc0-bdec0818d9b5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs" Jul 12 19:31:02.339332 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.339322 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1af07a3-642b-4014-8dc0-bdec0818d9b5-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs\" (UID: \"e1af07a3-642b-4014-8dc0-bdec0818d9b5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs" Jul 12 19:31:02.339622 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.339600 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1af07a3-642b-4014-8dc0-bdec0818d9b5-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs\" (UID: \"e1af07a3-642b-4014-8dc0-bdec0818d9b5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs" Jul 12 19:31:02.339701 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.339624 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1af07a3-642b-4014-8dc0-bdec0818d9b5-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs\" (UID: \"e1af07a3-642b-4014-8dc0-bdec0818d9b5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs" Jul 12 19:31:02.346361 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.346338 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsrz6\" (UniqueName: \"kubernetes.io/projected/e1af07a3-642b-4014-8dc0-bdec0818d9b5-kube-api-access-fsrz6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs\" (UID: \"e1af07a3-642b-4014-8dc0-bdec0818d9b5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs" Jul 12 19:31:02.466614 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.466555 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs" Jul 12 19:31:02.590857 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:02.590826 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs"] Jul 12 19:31:02.593626 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:31:02.593594 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1af07a3_642b_4014_8dc0_bdec0818d9b5.slice/crio-39d0283a3d60940f924e21f4109f2cd05dccc04f23d05f458f6b260fa1e6c400 WatchSource:0}: Error finding container 39d0283a3d60940f924e21f4109f2cd05dccc04f23d05f458f6b260fa1e6c400: Status 404 returned error can't find the container with id 39d0283a3d60940f924e21f4109f2cd05dccc04f23d05f458f6b260fa1e6c400 Jul 12 19:31:03.318388 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:03.318349 2564 generic.go:356] "Generic (PLEG): container finished" podID="3e8eb60c-253a-4201-b30c-561a435b44ab" containerID="507624889eb3a2291912f42b718b9993ae0271418bea166e87dcc72e0a6fb7d9" exitCode=0 Jul 12 19:31:03.318798 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:03.318425 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb" event={"ID":"3e8eb60c-253a-4201-b30c-561a435b44ab","Type":"ContainerDied","Data":"507624889eb3a2291912f42b718b9993ae0271418bea166e87dcc72e0a6fb7d9"} Jul 12 19:31:03.320240 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:03.320217 2564 generic.go:356] "Generic (PLEG): container finished" podID="673bd84f-85af-496f-b902-c74c065574fa" containerID="169105cee37084797a04021e708b0c354e9252eccbdc419b08b347c55d446400" exitCode=0 Jul 12 19:31:03.320357 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:03.320300 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc" event={"ID":"673bd84f-85af-496f-b902-c74c065574fa","Type":"ContainerDied","Data":"169105cee37084797a04021e708b0c354e9252eccbdc419b08b347c55d446400"} Jul 12 19:31:03.322240 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:03.322216 2564 generic.go:356] "Generic (PLEG): container finished" podID="a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67" containerID="a082f55e4bcdc4e843760f4477b48084d1f12f70e911437a54d1036b570923a9" exitCode=0 Jul 12 19:31:03.322318 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:03.322238 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl" event={"ID":"a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67","Type":"ContainerDied","Data":"a082f55e4bcdc4e843760f4477b48084d1f12f70e911437a54d1036b570923a9"} Jul 12 19:31:03.323816 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:03.323796 2564 generic.go:356] "Generic (PLEG): container finished" podID="e1af07a3-642b-4014-8dc0-bdec0818d9b5" containerID="acfe6a9f55cd2043182cf47ea4396681567c07cae8980bf72e6646d2c27ba761" exitCode=0 Jul 12 19:31:03.323905 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:03.323872 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs" event={"ID":"e1af07a3-642b-4014-8dc0-bdec0818d9b5","Type":"ContainerDied","Data":"acfe6a9f55cd2043182cf47ea4396681567c07cae8980bf72e6646d2c27ba761"} Jul 12 19:31:03.323960 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:03.323907 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs" event={"ID":"e1af07a3-642b-4014-8dc0-bdec0818d9b5","Type":"ContainerStarted","Data":"39d0283a3d60940f924e21f4109f2cd05dccc04f23d05f458f6b260fa1e6c400"} Jul 12 19:31:04.330436 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:04.330403 2564 generic.go:356] "Generic (PLEG): container finished" podID="673bd84f-85af-496f-b902-c74c065574fa" containerID="82b8719c6d9761fb285f28aac37446416cd5320a39466207868657adde87e271" exitCode=0 Jul 12 19:31:04.330850 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:04.330489 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc" event={"ID":"673bd84f-85af-496f-b902-c74c065574fa","Type":"ContainerDied","Data":"82b8719c6d9761fb285f28aac37446416cd5320a39466207868657adde87e271"} Jul 12 19:31:04.332463 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:04.332439 2564 generic.go:356] "Generic (PLEG): container finished" podID="a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67" containerID="4bc06a37aff7a10c2065ee4f8c4c0a7dde58aa300e4960cb13d7bc9ba70da15b" exitCode=0 Jul 12 19:31:04.332577 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:04.332518 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl" event={"ID":"a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67","Type":"ContainerDied","Data":"4bc06a37aff7a10c2065ee4f8c4c0a7dde58aa300e4960cb13d7bc9ba70da15b"} Jul 12 19:31:04.334079 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:04.334037 2564 generic.go:356] "Generic (PLEG): container finished" podID="e1af07a3-642b-4014-8dc0-bdec0818d9b5" containerID="00ce871ffd063e492ca10668dae965588747aab49371b913dae73dcacc12e541" exitCode=0 Jul 12 19:31:04.334171 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:04.334106 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs" event={"ID":"e1af07a3-642b-4014-8dc0-bdec0818d9b5","Type":"ContainerDied","Data":"00ce871ffd063e492ca10668dae965588747aab49371b913dae73dcacc12e541"} Jul 12 19:31:04.520651 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:04.520627 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb" Jul 12 19:31:04.658508 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:04.658473 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xm9l\" (UniqueName: \"kubernetes.io/projected/3e8eb60c-253a-4201-b30c-561a435b44ab-kube-api-access-8xm9l\") pod \"3e8eb60c-253a-4201-b30c-561a435b44ab\" (UID: \"3e8eb60c-253a-4201-b30c-561a435b44ab\") " Jul 12 19:31:04.658696 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:04.658533 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e8eb60c-253a-4201-b30c-561a435b44ab-bundle\") pod \"3e8eb60c-253a-4201-b30c-561a435b44ab\" (UID: \"3e8eb60c-253a-4201-b30c-561a435b44ab\") " Jul 12 19:31:04.658696 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:04.658570 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e8eb60c-253a-4201-b30c-561a435b44ab-util\") pod \"3e8eb60c-253a-4201-b30c-561a435b44ab\" (UID: \"3e8eb60c-253a-4201-b30c-561a435b44ab\") " Jul 12 19:31:04.659106 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:04.659072 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8eb60c-253a-4201-b30c-561a435b44ab-bundle" (OuterVolumeSpecName: "bundle") pod "3e8eb60c-253a-4201-b30c-561a435b44ab" (UID: "3e8eb60c-253a-4201-b30c-561a435b44ab"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:31:04.660731 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:04.660710 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8eb60c-253a-4201-b30c-561a435b44ab-kube-api-access-8xm9l" (OuterVolumeSpecName: "kube-api-access-8xm9l") pod "3e8eb60c-253a-4201-b30c-561a435b44ab" (UID: "3e8eb60c-253a-4201-b30c-561a435b44ab"). InnerVolumeSpecName "kube-api-access-8xm9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:31:04.663451 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:04.663427 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8eb60c-253a-4201-b30c-561a435b44ab-util" (OuterVolumeSpecName: "util") pod "3e8eb60c-253a-4201-b30c-561a435b44ab" (UID: "3e8eb60c-253a-4201-b30c-561a435b44ab"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:31:04.759198 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:04.759169 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xm9l\" (UniqueName: \"kubernetes.io/projected/3e8eb60c-253a-4201-b30c-561a435b44ab-kube-api-access-8xm9l\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:31:04.759198 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:04.759194 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e8eb60c-253a-4201-b30c-561a435b44ab-bundle\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:31:04.759198 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:04.759203 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e8eb60c-253a-4201-b30c-561a435b44ab-util\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:31:05.340985 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.340896 2564 generic.go:356] "Generic (PLEG): container finished" podID="e1af07a3-642b-4014-8dc0-bdec0818d9b5" containerID="97b739259faf1a6b2bc531d8da4b68b02745f615b031b99fe39df4d0b7ca5912" exitCode=0 Jul 12 19:31:05.341388 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.340986 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs" event={"ID":"e1af07a3-642b-4014-8dc0-bdec0818d9b5","Type":"ContainerDied","Data":"97b739259faf1a6b2bc531d8da4b68b02745f615b031b99fe39df4d0b7ca5912"} Jul 12 19:31:05.342813 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.342794 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb" Jul 12 19:31:05.342938 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.342826 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb" event={"ID":"3e8eb60c-253a-4201-b30c-561a435b44ab","Type":"ContainerDied","Data":"52a1355aa5fcd6618b00876befce6f9b536c11adaf96ac49aff20fcee17e2862"} Jul 12 19:31:05.342938 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.342862 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52a1355aa5fcd6618b00876befce6f9b536c11adaf96ac49aff20fcee17e2862" Jul 12 19:31:05.495944 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.495922 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl" Jul 12 19:31:05.498166 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.498143 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc" Jul 12 19:31:05.667263 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.667231 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jv74l\" (UniqueName: \"kubernetes.io/projected/673bd84f-85af-496f-b902-c74c065574fa-kube-api-access-jv74l\") pod \"673bd84f-85af-496f-b902-c74c065574fa\" (UID: \"673bd84f-85af-496f-b902-c74c065574fa\") " Jul 12 19:31:05.667459 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.667293 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gh95q\" (UniqueName: \"kubernetes.io/projected/a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67-kube-api-access-gh95q\") pod \"a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67\" (UID: \"a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67\") " Jul 12 19:31:05.667459 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.667336 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/673bd84f-85af-496f-b902-c74c065574fa-bundle\") pod \"673bd84f-85af-496f-b902-c74c065574fa\" (UID: \"673bd84f-85af-496f-b902-c74c065574fa\") " Jul 12 19:31:05.667459 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.667396 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67-bundle\") pod \"a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67\" (UID: \"a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67\") " Jul 12 19:31:05.667459 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.667424 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/673bd84f-85af-496f-b902-c74c065574fa-util\") pod \"673bd84f-85af-496f-b902-c74c065574fa\" (UID: \"673bd84f-85af-496f-b902-c74c065574fa\") " Jul 12 19:31:05.667667 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.667484 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67-util\") pod \"a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67\" (UID: \"a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67\") " Jul 12 19:31:05.668068 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.668011 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673bd84f-85af-496f-b902-c74c065574fa-bundle" (OuterVolumeSpecName: "bundle") pod "673bd84f-85af-496f-b902-c74c065574fa" (UID: "673bd84f-85af-496f-b902-c74c065574fa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:31:05.668243 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.668204 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67-bundle" (OuterVolumeSpecName: "bundle") pod "a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67" (UID: "a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:31:05.669664 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.669634 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67-kube-api-access-gh95q" (OuterVolumeSpecName: "kube-api-access-gh95q") pod "a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67" (UID: "a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67"). InnerVolumeSpecName "kube-api-access-gh95q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:31:05.670113 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.670086 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673bd84f-85af-496f-b902-c74c065574fa-kube-api-access-jv74l" (OuterVolumeSpecName: "kube-api-access-jv74l") pod "673bd84f-85af-496f-b902-c74c065574fa" (UID: "673bd84f-85af-496f-b902-c74c065574fa"). InnerVolumeSpecName "kube-api-access-jv74l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:31:05.673165 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.673133 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673bd84f-85af-496f-b902-c74c065574fa-util" (OuterVolumeSpecName: "util") pod "673bd84f-85af-496f-b902-c74c065574fa" (UID: "673bd84f-85af-496f-b902-c74c065574fa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:31:05.675605 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.675579 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67-util" (OuterVolumeSpecName: "util") pod "a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67" (UID: "a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:31:05.768723 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.768673 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67-util\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:31:05.768723 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.768715 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv74l\" (UniqueName: \"kubernetes.io/projected/673bd84f-85af-496f-b902-c74c065574fa-kube-api-access-jv74l\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:31:05.768723 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.768726 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh95q\" (UniqueName: \"kubernetes.io/projected/a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67-kube-api-access-gh95q\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:31:05.768723 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.768735 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/673bd84f-85af-496f-b902-c74c065574fa-bundle\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:31:05.768723 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.768743 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67-bundle\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:31:05.769002 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:05.768750 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/673bd84f-85af-496f-b902-c74c065574fa-util\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:31:06.349674 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:06.349583 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc" event={"ID":"673bd84f-85af-496f-b902-c74c065574fa","Type":"ContainerDied","Data":"cea60a8bc09f96b352b064be554db1425715268bfe78a37e1a03785f47fe8ae3"} Jul 12 19:31:06.349674 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:06.349661 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cea60a8bc09f96b352b064be554db1425715268bfe78a37e1a03785f47fe8ae3" Jul 12 19:31:06.349674 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:06.349606 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc" Jul 12 19:31:06.351685 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:06.351574 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl" event={"ID":"a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67","Type":"ContainerDied","Data":"ec65b33129ce77c05f55c19436b1ef72d6ebdd147c3bda6a718a9307f7f90f4a"} Jul 12 19:31:06.351685 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:06.351632 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec65b33129ce77c05f55c19436b1ef72d6ebdd147c3bda6a718a9307f7f90f4a" Jul 12 19:31:06.351890 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:06.351790 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl" Jul 12 19:31:06.485407 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:06.485385 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs" Jul 12 19:31:06.575528 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:06.575484 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsrz6\" (UniqueName: \"kubernetes.io/projected/e1af07a3-642b-4014-8dc0-bdec0818d9b5-kube-api-access-fsrz6\") pod \"e1af07a3-642b-4014-8dc0-bdec0818d9b5\" (UID: \"e1af07a3-642b-4014-8dc0-bdec0818d9b5\") " Jul 12 19:31:06.575696 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:06.575603 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1af07a3-642b-4014-8dc0-bdec0818d9b5-bundle\") pod \"e1af07a3-642b-4014-8dc0-bdec0818d9b5\" (UID: \"e1af07a3-642b-4014-8dc0-bdec0818d9b5\") " Jul 12 19:31:06.575696 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:06.575639 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1af07a3-642b-4014-8dc0-bdec0818d9b5-util\") pod \"e1af07a3-642b-4014-8dc0-bdec0818d9b5\" (UID: \"e1af07a3-642b-4014-8dc0-bdec0818d9b5\") " Jul 12 19:31:06.576217 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:06.576185 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1af07a3-642b-4014-8dc0-bdec0818d9b5-bundle" (OuterVolumeSpecName: "bundle") pod "e1af07a3-642b-4014-8dc0-bdec0818d9b5" (UID: "e1af07a3-642b-4014-8dc0-bdec0818d9b5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:31:06.578091 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:06.578056 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1af07a3-642b-4014-8dc0-bdec0818d9b5-kube-api-access-fsrz6" (OuterVolumeSpecName: "kube-api-access-fsrz6") pod "e1af07a3-642b-4014-8dc0-bdec0818d9b5" (UID: "e1af07a3-642b-4014-8dc0-bdec0818d9b5"). InnerVolumeSpecName "kube-api-access-fsrz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:31:06.581270 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:06.581244 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1af07a3-642b-4014-8dc0-bdec0818d9b5-util" (OuterVolumeSpecName: "util") pod "e1af07a3-642b-4014-8dc0-bdec0818d9b5" (UID: "e1af07a3-642b-4014-8dc0-bdec0818d9b5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:31:06.676432 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:06.676398 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1af07a3-642b-4014-8dc0-bdec0818d9b5-bundle\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:31:06.676432 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:06.676427 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1af07a3-642b-4014-8dc0-bdec0818d9b5-util\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:31:06.676432 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:06.676436 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsrz6\" (UniqueName: \"kubernetes.io/projected/e1af07a3-642b-4014-8dc0-bdec0818d9b5-kube-api-access-fsrz6\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:31:07.358464 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:07.358429 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs" event={"ID":"e1af07a3-642b-4014-8dc0-bdec0818d9b5","Type":"ContainerDied","Data":"39d0283a3d60940f924e21f4109f2cd05dccc04f23d05f458f6b260fa1e6c400"} Jul 12 19:31:07.358464 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:07.358458 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs" Jul 12 19:31:07.358877 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:07.358475 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39d0283a3d60940f924e21f4109f2cd05dccc04f23d05f458f6b260fa1e6c400" Jul 12 19:31:14.120710 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:14.120677 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:16.811232 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811198 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk"] Jul 12 19:31:16.811584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811477 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e8eb60c-253a-4201-b30c-561a435b44ab" containerName="pull" Jul 12 19:31:16.811584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811486 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e8eb60c-253a-4201-b30c-561a435b44ab" containerName="pull" Jul 12 19:31:16.811584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811495 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1af07a3-642b-4014-8dc0-bdec0818d9b5" containerName="pull" Jul 12 19:31:16.811584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811501 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1af07a3-642b-4014-8dc0-bdec0818d9b5" containerName="pull" Jul 12 19:31:16.811584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811511 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e8eb60c-253a-4201-b30c-561a435b44ab" containerName="extract" Jul 12 19:31:16.811584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811516 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e8eb60c-253a-4201-b30c-561a435b44ab" containerName="extract" Jul 12 19:31:16.811584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811522 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673bd84f-85af-496f-b902-c74c065574fa" containerName="extract" Jul 12 19:31:16.811584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811527 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="673bd84f-85af-496f-b902-c74c065574fa" containerName="extract" Jul 12 19:31:16.811584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811533 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67" containerName="pull" Jul 12 19:31:16.811584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811538 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67" containerName="pull" Jul 12 19:31:16.811584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811542 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673bd84f-85af-496f-b902-c74c065574fa" containerName="pull" Jul 12 19:31:16.811584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811547 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="673bd84f-85af-496f-b902-c74c065574fa" containerName="pull" Jul 12 19:31:16.811584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811554 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1af07a3-642b-4014-8dc0-bdec0818d9b5" containerName="util" Jul 12 19:31:16.811584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811560 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1af07a3-642b-4014-8dc0-bdec0818d9b5" containerName="util" Jul 12 19:31:16.811584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811565 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67" containerName="extract" Jul 12 19:31:16.811584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811570 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67" containerName="extract" Jul 12 19:31:16.811584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811575 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e8eb60c-253a-4201-b30c-561a435b44ab" containerName="util" Jul 12 19:31:16.811584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811580 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e8eb60c-253a-4201-b30c-561a435b44ab" containerName="util" Jul 12 19:31:16.811584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811585 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673bd84f-85af-496f-b902-c74c065574fa" containerName="util" Jul 12 19:31:16.811584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811592 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="673bd84f-85af-496f-b902-c74c065574fa" containerName="util" Jul 12 19:31:16.812181 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811597 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1af07a3-642b-4014-8dc0-bdec0818d9b5" containerName="extract" Jul 12 19:31:16.812181 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811602 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1af07a3-642b-4014-8dc0-bdec0818d9b5" containerName="extract" Jul 12 19:31:16.812181 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811609 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67" containerName="util" Jul 12 19:31:16.812181 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811613 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67" containerName="util" Jul 12 19:31:16.812181 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811664 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="673bd84f-85af-496f-b902-c74c065574fa" containerName="extract" Jul 12 19:31:16.812181 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811672 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1af07a3-642b-4014-8dc0-bdec0818d9b5" containerName="extract" Jul 12 19:31:16.812181 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811678 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67" containerName="extract" Jul 12 19:31:16.812181 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.811683 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e8eb60c-253a-4201-b30c-561a435b44ab" containerName="extract" Jul 12 19:31:16.817679 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.817657 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk" Jul 12 19:31:16.820875 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.820854 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8db77\"" Jul 12 19:31:16.833318 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.833294 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk"] Jul 12 19:31:16.953541 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:16.953502 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wmdv\" (UniqueName: \"kubernetes.io/projected/5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3-kube-api-access-9wmdv\") pod \"limitador-operator-controller-manager-859d7b78bd-m9vxk\" (UID: \"5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk" Jul 12 19:31:17.053992 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:17.053960 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wmdv\" (UniqueName: \"kubernetes.io/projected/5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3-kube-api-access-9wmdv\") pod \"limitador-operator-controller-manager-859d7b78bd-m9vxk\" (UID: \"5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk" Jul 12 19:31:17.071528 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:17.071457 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wmdv\" (UniqueName: \"kubernetes.io/projected/5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3-kube-api-access-9wmdv\") pod \"limitador-operator-controller-manager-859d7b78bd-m9vxk\" (UID: \"5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk" Jul 12 19:31:17.133174 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:17.133146 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk" Jul 12 19:31:17.264693 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:17.264665 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk"] Jul 12 19:31:17.267176 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:31:17.267146 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5965ad9f_2a6d_46d0_a7e8_36d837d7a7f3.slice/crio-9abbdcda0bfa9e1c1ad7490a4f37bd2bb12bbadd753d809fe35aa75afc5d79b4 WatchSource:0}: Error finding container 9abbdcda0bfa9e1c1ad7490a4f37bd2bb12bbadd753d809fe35aa75afc5d79b4: Status 404 returned error can't find the container with id 9abbdcda0bfa9e1c1ad7490a4f37bd2bb12bbadd753d809fe35aa75afc5d79b4 Jul 12 19:31:17.269116 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:17.269093 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 19:31:17.404160 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:17.404123 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk" event={"ID":"5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3","Type":"ContainerStarted","Data":"9abbdcda0bfa9e1c1ad7490a4f37bd2bb12bbadd753d809fe35aa75afc5d79b4"} Jul 12 19:31:17.725008 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:17.724917 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:19.415996 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:19.415960 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk" event={"ID":"5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3","Type":"ContainerStarted","Data":"b9b0f1fd23ca33969e50a5306194756a354b8d56ef2f5c7f6603633318a3a10d"} Jul 12 19:31:19.416391 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:19.416046 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk" Jul 12 19:31:19.432412 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:19.432362 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk" podStartSLOduration=2.015866563 podStartE2EDuration="3.432345972s" totalImagesPullingTime="1.416479409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:31:19.430694634 +0000 UTC m=+688.023968731" watchObservedRunningTime="2026-07-12 19:31:19.432345972 +0000 UTC m=+688.025620071" Jul 12 19:31:21.713004 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:21.712972 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z"] Jul 12 19:31:21.718113 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:21.718091 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" Jul 12 19:31:21.720482 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:21.720463 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kcwqf\"" Jul 12 19:31:21.729871 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:21.729849 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z"] Jul 12 19:31:21.791350 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:21.791314 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csrdk\" (UniqueName: \"kubernetes.io/projected/490ff7fd-bf14-402c-84f8-beb2ddc2ba7f-kube-api-access-csrdk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7cm7z\" (UID: \"490ff7fd-bf14-402c-84f8-beb2ddc2ba7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" Jul 12 19:31:21.791350 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:21.791353 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/490ff7fd-bf14-402c-84f8-beb2ddc2ba7f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7cm7z\" (UID: \"490ff7fd-bf14-402c-84f8-beb2ddc2ba7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" Jul 12 19:31:21.891752 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:21.891714 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csrdk\" (UniqueName: \"kubernetes.io/projected/490ff7fd-bf14-402c-84f8-beb2ddc2ba7f-kube-api-access-csrdk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7cm7z\" (UID: \"490ff7fd-bf14-402c-84f8-beb2ddc2ba7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" Jul 12 19:31:21.891752 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:21.891753 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/490ff7fd-bf14-402c-84f8-beb2ddc2ba7f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7cm7z\" (UID: \"490ff7fd-bf14-402c-84f8-beb2ddc2ba7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" Jul 12 19:31:21.892186 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:21.892166 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/490ff7fd-bf14-402c-84f8-beb2ddc2ba7f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7cm7z\" (UID: \"490ff7fd-bf14-402c-84f8-beb2ddc2ba7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" Jul 12 19:31:21.907012 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:21.906983 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csrdk\" (UniqueName: \"kubernetes.io/projected/490ff7fd-bf14-402c-84f8-beb2ddc2ba7f-kube-api-access-csrdk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7cm7z\" (UID: \"490ff7fd-bf14-402c-84f8-beb2ddc2ba7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" Jul 12 19:31:22.028300 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:22.028216 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" Jul 12 19:31:22.123513 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:22.123465 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:22.169975 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:22.169953 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z"] Jul 12 19:31:22.172206 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:31:22.172180 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490ff7fd_bf14_402c_84f8_beb2ddc2ba7f.slice/crio-b290b0cc6867457a46c453295b566396ea4633398c458e0507399f47091d8d2f WatchSource:0}: Error finding container b290b0cc6867457a46c453295b566396ea4633398c458e0507399f47091d8d2f: Status 404 returned error can't find the container with id b290b0cc6867457a46c453295b566396ea4633398c458e0507399f47091d8d2f Jul 12 19:31:22.431736 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:22.431707 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" event={"ID":"490ff7fd-bf14-402c-84f8-beb2ddc2ba7f","Type":"ContainerStarted","Data":"b290b0cc6867457a46c453295b566396ea4633398c458e0507399f47091d8d2f"} Jul 12 19:31:22.723497 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:22.723369 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:23.193539 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:23.193505 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:24.271281 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:24.271243 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:24.331146 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:24.331113 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:24.953774 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:24.953735 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:25.312091 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:25.311983 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-6zcqm"] Jul 12 19:31:25.315679 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:25.315657 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6zcqm" Jul 12 19:31:25.317939 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:25.317919 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-sq42q\"" Jul 12 19:31:25.324743 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:25.324722 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-6zcqm"] Jul 12 19:31:25.422331 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:25.422281 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlwjn\" (UniqueName: \"kubernetes.io/projected/7bae773d-e2a2-4e14-86c1-3822c2223fd5-kube-api-access-zlwjn\") pod \"authorino-operator-75cc8d5cb7-6zcqm\" (UID: \"7bae773d-e2a2-4e14-86c1-3822c2223fd5\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-6zcqm" Jul 12 19:31:25.523216 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:25.523175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlwjn\" (UniqueName: \"kubernetes.io/projected/7bae773d-e2a2-4e14-86c1-3822c2223fd5-kube-api-access-zlwjn\") pod \"authorino-operator-75cc8d5cb7-6zcqm\" (UID: \"7bae773d-e2a2-4e14-86c1-3822c2223fd5\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-6zcqm" Jul 12 19:31:25.535689 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:25.535660 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlwjn\" (UniqueName: \"kubernetes.io/projected/7bae773d-e2a2-4e14-86c1-3822c2223fd5-kube-api-access-zlwjn\") pod \"authorino-operator-75cc8d5cb7-6zcqm\" (UID: \"7bae773d-e2a2-4e14-86c1-3822c2223fd5\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-6zcqm" Jul 12 19:31:25.626397 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:25.626292 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6zcqm" Jul 12 19:31:25.773102 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:25.773076 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-6zcqm"] Jul 12 19:31:25.775318 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:31:25.775291 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bae773d_e2a2_4e14_86c1_3822c2223fd5.slice/crio-752ce2281d9d6cdc88e1876b9474f94f788ed26639e9da7bedffbcd29dcdb773 WatchSource:0}: Error finding container 752ce2281d9d6cdc88e1876b9474f94f788ed26639e9da7bedffbcd29dcdb773: Status 404 returned error can't find the container with id 752ce2281d9d6cdc88e1876b9474f94f788ed26639e9da7bedffbcd29dcdb773 Jul 12 19:31:26.452514 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:26.452471 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6zcqm" event={"ID":"7bae773d-e2a2-4e14-86c1-3822c2223fd5","Type":"ContainerStarted","Data":"752ce2281d9d6cdc88e1876b9474f94f788ed26639e9da7bedffbcd29dcdb773"} Jul 12 19:31:26.525949 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:26.525847 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:27.564710 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:27.564671 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:29.469382 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:29.469346 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6zcqm" event={"ID":"7bae773d-e2a2-4e14-86c1-3822c2223fd5","Type":"ContainerStarted","Data":"b02801f3b1de5b390e7c75c2906b6e0e43bcbcaae64214a3dddbfad26406fddc"} Jul 12 19:31:29.469812 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:29.469428 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6zcqm" Jul 12 19:31:29.470986 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:29.470960 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" event={"ID":"490ff7fd-bf14-402c-84f8-beb2ddc2ba7f","Type":"ContainerStarted","Data":"09168c440a24eede84e111ae47574500837876fe9f621f8b0797ecfb4693da43"} Jul 12 19:31:29.471120 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:29.471058 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" Jul 12 19:31:29.487807 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:29.487759 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6zcqm" podStartSLOduration=1.293870083 podStartE2EDuration="4.487740909s" totalImagesPullingTime="3.193870826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:31:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:31:29.486527234 +0000 UTC m=+698.079801335" watchObservedRunningTime="2026-07-12 19:31:29.487740909 +0000 UTC m=+698.081015008" Jul 12 19:31:29.516038 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:29.515967 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" podStartSLOduration=1.754469296 podStartE2EDuration="8.515951955s" totalImagesPullingTime="6.761482659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:31:29.514868371 +0000 UTC m=+698.108142470" watchObservedRunningTime="2026-07-12 19:31:29.515951955 +0000 UTC m=+698.109226053" Jul 12 19:31:29.549092 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:29.549060 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:30.423892 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:30.423862 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk" Jul 12 19:31:30.950815 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:30.950778 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:31.117553 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:31.117505 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:31.752156 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:31.752122 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pmvq9"] Jul 12 19:31:31.760833 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:31.760808 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pmvq9" Jul 12 19:31:31.763413 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:31.763384 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 12 19:31:31.763569 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:31.763488 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 12 19:31:31.763569 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:31.763542 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qlwpj\"" Jul 12 19:31:31.764172 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:31.764149 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pmvq9"] Jul 12 19:31:31.765858 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:31.765834 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:31.774529 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:31.774509 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n28h\" (UniqueName: \"kubernetes.io/projected/ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9-kube-api-access-9n28h\") pod \"kuadrant-console-plugin-66b5774985-pmvq9\" (UID: \"ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pmvq9" Jul 12 19:31:31.774619 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:31.774560 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pmvq9\" (UID: \"ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pmvq9" Jul 12 19:31:31.774691 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:31.774641 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pmvq9\" (UID: \"ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pmvq9" Jul 12 19:31:31.870155 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:31.870115 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:31.875050 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:31.875013 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pmvq9\" (UID: \"ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pmvq9" Jul 12 19:31:31.875128 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:31.875071 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n28h\" (UniqueName: \"kubernetes.io/projected/ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9-kube-api-access-9n28h\") pod \"kuadrant-console-plugin-66b5774985-pmvq9\" (UID: \"ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pmvq9" Jul 12 19:31:31.875162 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:31.875128 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pmvq9\" (UID: \"ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pmvq9" Jul 12 19:31:31.875791 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:31.875772 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pmvq9\" (UID: \"ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pmvq9" Jul 12 19:31:31.877531 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:31.877515 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pmvq9\" (UID: \"ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pmvq9" Jul 12 19:31:31.882700 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:31.882674 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n28h\" (UniqueName: \"kubernetes.io/projected/ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9-kube-api-access-9n28h\") pod \"kuadrant-console-plugin-66b5774985-pmvq9\" (UID: \"ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pmvq9" Jul 12 19:31:32.071810 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:32.071731 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pmvq9" Jul 12 19:31:32.230036 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:32.229990 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pmvq9"] Jul 12 19:31:32.232931 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:31:32.232903 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce2f6ec1_e4cf_4a36_a39d_662a3c970ed9.slice/crio-53ff834a0d2f4f4b1d0d1b241e69f0e7f7585949743e764395cdc4cf64705d37 WatchSource:0}: Error finding container 53ff834a0d2f4f4b1d0d1b241e69f0e7f7585949743e764395cdc4cf64705d37: Status 404 returned error can't find the container with id 53ff834a0d2f4f4b1d0d1b241e69f0e7f7585949743e764395cdc4cf64705d37 Jul 12 19:31:32.486085 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:32.486047 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pmvq9" event={"ID":"ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9","Type":"ContainerStarted","Data":"53ff834a0d2f4f4b1d0d1b241e69f0e7f7585949743e764395cdc4cf64705d37"} Jul 12 19:31:34.126444 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:34.126407 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:34.177356 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:34.177321 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:34.888175 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:34.888138 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:34.989192 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:34.989153 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:35.090474 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:35.090437 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:40.478650 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:40.478613 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" Jul 12 19:31:40.479098 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:40.478784 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6zcqm" Jul 12 19:31:42.280886 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.280670 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z"] Jul 12 19:31:42.281335 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.280883 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" podUID="490ff7fd-bf14-402c-84f8-beb2ddc2ba7f" containerName="manager" containerID="cri-o://09168c440a24eede84e111ae47574500837876fe9f621f8b0797ecfb4693da43" gracePeriod=2 Jul 12 19:31:42.316966 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.316898 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z"] Jul 12 19:31:42.346796 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:31:42.346744 2564 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"490ff7fd-bf14-402c-84f8-beb2ddc2ba7f\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-7cm7z\": pods \"kuadrant-operator-controller-manager-8684c9ff49-7cm7z\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" Jul 12 19:31:42.390437 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.390402 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk"] Jul 12 19:31:42.390674 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.390644 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk" podUID="5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3" containerName="manager" containerID="cri-o://b9b0f1fd23ca33969e50a5306194756a354b8d56ef2f5c7f6603633318a3a10d" gracePeriod=2 Jul 12 19:31:42.395690 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.395664 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z"] Jul 12 19:31:42.395973 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.395961 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="490ff7fd-bf14-402c-84f8-beb2ddc2ba7f" containerName="manager" Jul 12 19:31:42.395973 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.395974 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="490ff7fd-bf14-402c-84f8-beb2ddc2ba7f" containerName="manager" Jul 12 19:31:42.396105 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.396079 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="490ff7fd-bf14-402c-84f8-beb2ddc2ba7f" containerName="manager" Jul 12 19:31:42.396841 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:31:42.396716 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7cm7z\" is forbidden: User \"system:node:ip-10-0-130-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-167.ec2.internal' and this object" podUID="490ff7fd-bf14-402c-84f8-beb2ddc2ba7f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" Jul 12 19:31:42.403498 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.403473 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z" Jul 12 19:31:42.405508 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.405477 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk"] Jul 12 19:31:42.432870 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.432218 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z"] Jul 12 19:31:42.442335 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:31:42.442305 2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5965ad9f_2a6d_46d0_a7e8_36d837d7a7f3.slice/crio-conmon-b9b0f1fd23ca33969e50a5306194756a354b8d56ef2f5c7f6603633318a3a10d.scope\": RecentStats: unable to find data in memory cache]" Jul 12 19:31:42.448656 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.448625 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kxblk"] Jul 12 19:31:42.449135 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.449117 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3" containerName="manager" Jul 12 19:31:42.449135 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.449136 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3" containerName="manager" Jul 12 19:31:42.449275 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.449241 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3" containerName="manager" Jul 12 19:31:42.454445 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.454422 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kxblk" Jul 12 19:31:42.469685 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:31:42.469603 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7cm7z\" is forbidden: User \"system:node:ip-10-0-130-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-167.ec2.internal' and this object" podUID="490ff7fd-bf14-402c-84f8-beb2ddc2ba7f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" Jul 12 19:31:42.470711 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.470684 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kxblk"] Jul 12 19:31:42.472012 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:31:42.471983 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-m9vxk\" is forbidden: User \"system:node:ip-10-0-130-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-167.ec2.internal' and this object" podUID="5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk" Jul 12 19:31:42.472898 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.472868 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz8bv\" (UniqueName: \"kubernetes.io/projected/e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4-kube-api-access-qz8bv\") pod \"kuadrant-operator-controller-manager-86878c769f-wbc8z\" (UID: \"e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z" Jul 12 19:31:42.473040 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.472924 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wbc8z\" (UID: \"e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z" Jul 12 19:31:42.474335 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:31:42.474231 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7cm7z\" is forbidden: User \"system:node:ip-10-0-130-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-167.ec2.internal' and this object" podUID="490ff7fd-bf14-402c-84f8-beb2ddc2ba7f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" Jul 12 19:31:42.574333 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.574251 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk8pj\" (UniqueName: \"kubernetes.io/projected/be6cf8a0-8822-41f2-a079-905b10525b94-kube-api-access-hk8pj\") pod \"limitador-operator-controller-manager-859d7b78bd-kxblk\" (UID: \"be6cf8a0-8822-41f2-a079-905b10525b94\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kxblk" Jul 12 19:31:42.574506 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.574355 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz8bv\" (UniqueName: \"kubernetes.io/projected/e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4-kube-api-access-qz8bv\") pod \"kuadrant-operator-controller-manager-86878c769f-wbc8z\" (UID: \"e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z" Jul 12 19:31:42.574506 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.574406 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wbc8z\" (UID: \"e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z" Jul 12 19:31:42.574779 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.574758 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wbc8z\" (UID: \"e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z" Jul 12 19:31:42.589626 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.589595 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz8bv\" (UniqueName: \"kubernetes.io/projected/e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4-kube-api-access-qz8bv\") pod \"kuadrant-operator-controller-manager-86878c769f-wbc8z\" (UID: \"e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z" Jul 12 19:31:42.675660 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.675621 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk8pj\" (UniqueName: \"kubernetes.io/projected/be6cf8a0-8822-41f2-a079-905b10525b94-kube-api-access-hk8pj\") pod \"limitador-operator-controller-manager-859d7b78bd-kxblk\" (UID: \"be6cf8a0-8822-41f2-a079-905b10525b94\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kxblk" Jul 12 19:31:42.684617 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.684587 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk8pj\" (UniqueName: \"kubernetes.io/projected/be6cf8a0-8822-41f2-a079-905b10525b94-kube-api-access-hk8pj\") pod \"limitador-operator-controller-manager-859d7b78bd-kxblk\" (UID: \"be6cf8a0-8822-41f2-a079-905b10525b94\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kxblk" Jul 12 19:31:42.779475 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.779438 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z" Jul 12 19:31:42.787568 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.787537 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kxblk" Jul 12 19:31:42.968068 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:42.968008 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:43.065064 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:43.065002 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:45.380827 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:45.380778 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:45.480758 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:45.480715 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:49.722999 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:49.722963 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:49.823208 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:49.823171 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:50.368714 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:50.368672 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:51.933581 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:51.933531 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:51.983164 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:51.983126 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:53.222250 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.222138 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" Jul 12 19:31:53.224530 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:31:53.224361 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7cm7z\" is forbidden: User \"system:node:ip-10-0-130-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-167.ec2.internal' and this object" podUID="490ff7fd-bf14-402c-84f8-beb2ddc2ba7f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" Jul 12 19:31:53.226479 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.226458 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z"] Jul 12 19:31:53.229706 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:31:53.229668 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0634f8b_aaf5_4e41_b384_a8c9a2b2a4c4.slice/crio-0ccfabe52e431b2107d50558a911760cb24eff670ff9348953e3eb582c9eaf93 WatchSource:0}: Error finding container 0ccfabe52e431b2107d50558a911760cb24eff670ff9348953e3eb582c9eaf93: Status 404 returned error can't find the container with id 0ccfabe52e431b2107d50558a911760cb24eff670ff9348953e3eb582c9eaf93 Jul 12 19:31:53.234121 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.234100 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk" Jul 12 19:31:53.235977 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:31:53.235955 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7cm7z\" is forbidden: User \"system:node:ip-10-0-130-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-167.ec2.internal' and this object" podUID="490ff7fd-bf14-402c-84f8-beb2ddc2ba7f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" Jul 12 19:31:53.237965 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:31:53.237939 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-m9vxk\" is forbidden: User \"system:node:ip-10-0-130-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-167.ec2.internal' and this object" podUID="5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk" Jul 12 19:31:53.254258 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.254195 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kxblk"] Jul 12 19:31:53.267390 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:31:53.267359 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6cf8a0_8822_41f2_a079_905b10525b94.slice/crio-ef063351ebd52ee7476047763907a3ec0cd9f2b74b06092fb4b9007867cd8e63 WatchSource:0}: Error finding container ef063351ebd52ee7476047763907a3ec0cd9f2b74b06092fb4b9007867cd8e63: Status 404 returned error can't find the container with id ef063351ebd52ee7476047763907a3ec0cd9f2b74b06092fb4b9007867cd8e63 Jul 12 19:31:53.376895 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.376862 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wmdv\" (UniqueName: \"kubernetes.io/projected/5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3-kube-api-access-9wmdv\") pod \"5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3\" (UID: \"5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3\") " Jul 12 19:31:53.377109 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.376913 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/490ff7fd-bf14-402c-84f8-beb2ddc2ba7f-extensions-socket-volume\") pod \"490ff7fd-bf14-402c-84f8-beb2ddc2ba7f\" (UID: \"490ff7fd-bf14-402c-84f8-beb2ddc2ba7f\") " Jul 12 19:31:53.377109 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.376933 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csrdk\" (UniqueName: \"kubernetes.io/projected/490ff7fd-bf14-402c-84f8-beb2ddc2ba7f-kube-api-access-csrdk\") pod \"490ff7fd-bf14-402c-84f8-beb2ddc2ba7f\" (UID: \"490ff7fd-bf14-402c-84f8-beb2ddc2ba7f\") " Jul 12 19:31:53.377467 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.377440 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/490ff7fd-bf14-402c-84f8-beb2ddc2ba7f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "490ff7fd-bf14-402c-84f8-beb2ddc2ba7f" (UID: "490ff7fd-bf14-402c-84f8-beb2ddc2ba7f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:31:53.379159 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.379138 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3-kube-api-access-9wmdv" (OuterVolumeSpecName: "kube-api-access-9wmdv") pod "5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3" (UID: "5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3"). InnerVolumeSpecName "kube-api-access-9wmdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:31:53.379253 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.379183 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490ff7fd-bf14-402c-84f8-beb2ddc2ba7f-kube-api-access-csrdk" (OuterVolumeSpecName: "kube-api-access-csrdk") pod "490ff7fd-bf14-402c-84f8-beb2ddc2ba7f" (UID: "490ff7fd-bf14-402c-84f8-beb2ddc2ba7f"). InnerVolumeSpecName "kube-api-access-csrdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:31:53.477882 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.477848 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wmdv\" (UniqueName: \"kubernetes.io/projected/5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3-kube-api-access-9wmdv\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:31:53.477882 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.477877 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/490ff7fd-bf14-402c-84f8-beb2ddc2ba7f-extensions-socket-volume\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:31:53.477882 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.477887 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csrdk\" (UniqueName: \"kubernetes.io/projected/490ff7fd-bf14-402c-84f8-beb2ddc2ba7f-kube-api-access-csrdk\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:31:53.605597 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.605511 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pmvq9" event={"ID":"ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9","Type":"ContainerStarted","Data":"cfc1fb0e6e1ef745e3235cf320261a00379453e9c6964ba7de49f6f00ddc4560"} Jul 12 19:31:53.606834 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.606812 2564 generic.go:356] "Generic (PLEG): container finished" podID="490ff7fd-bf14-402c-84f8-beb2ddc2ba7f" containerID="09168c440a24eede84e111ae47574500837876fe9f621f8b0797ecfb4693da43" exitCode=0 Jul 12 19:31:53.606968 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.606872 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" Jul 12 19:31:53.606968 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.606911 2564 scope.go:117] "RemoveContainer" containerID="09168c440a24eede84e111ae47574500837876fe9f621f8b0797ecfb4693da43" Jul 12 19:31:53.607829 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:31:53.607798 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-m9vxk\" is forbidden: User \"system:node:ip-10-0-130-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-167.ec2.internal' and this object" podUID="5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk" Jul 12 19:31:53.608524 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.608493 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z" event={"ID":"e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4","Type":"ContainerStarted","Data":"5076d6ffe93c4942d3fe2fe03366a291e257dfb754abfb88f198f886f50507ac"} Jul 12 19:31:53.608619 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.608536 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z" event={"ID":"e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4","Type":"ContainerStarted","Data":"0ccfabe52e431b2107d50558a911760cb24eff670ff9348953e3eb582c9eaf93"} Jul 12 19:31:53.608619 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.608589 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z" Jul 12 19:31:53.609668 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:31:53.609641 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7cm7z\" is forbidden: User \"system:node:ip-10-0-130-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-167.ec2.internal' and this object" podUID="490ff7fd-bf14-402c-84f8-beb2ddc2ba7f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" Jul 12 19:31:53.610297 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.610269 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kxblk" event={"ID":"be6cf8a0-8822-41f2-a079-905b10525b94","Type":"ContainerStarted","Data":"56432763aec97624ae05a6d8c94a38963b16f262ae619676172d649872444667"} Jul 12 19:31:53.610384 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.610343 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kxblk" Jul 12 19:31:53.610384 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.610365 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kxblk" event={"ID":"be6cf8a0-8822-41f2-a079-905b10525b94","Type":"ContainerStarted","Data":"ef063351ebd52ee7476047763907a3ec0cd9f2b74b06092fb4b9007867cd8e63"} Jul 12 19:31:53.611710 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.611690 2564 generic.go:356] "Generic (PLEG): container finished" podID="5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3" containerID="b9b0f1fd23ca33969e50a5306194756a354b8d56ef2f5c7f6603633318a3a10d" exitCode=0 Jul 12 19:31:53.611803 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.611749 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk" Jul 12 19:31:53.618103 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.618085 2564 scope.go:117] "RemoveContainer" containerID="09168c440a24eede84e111ae47574500837876fe9f621f8b0797ecfb4693da43" Jul 12 19:31:53.618397 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:31:53.618362 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09168c440a24eede84e111ae47574500837876fe9f621f8b0797ecfb4693da43\": container with ID starting with 09168c440a24eede84e111ae47574500837876fe9f621f8b0797ecfb4693da43 not found: ID does not exist" containerID="09168c440a24eede84e111ae47574500837876fe9f621f8b0797ecfb4693da43" Jul 12 19:31:53.618478 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.618410 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09168c440a24eede84e111ae47574500837876fe9f621f8b0797ecfb4693da43"} err="failed to get container status \"09168c440a24eede84e111ae47574500837876fe9f621f8b0797ecfb4693da43\": rpc error: code = NotFound desc = could not find container \"09168c440a24eede84e111ae47574500837876fe9f621f8b0797ecfb4693da43\": container with ID starting with 09168c440a24eede84e111ae47574500837876fe9f621f8b0797ecfb4693da43 not found: ID does not exist" Jul 12 19:31:53.618478 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.618433 2564 scope.go:117] "RemoveContainer" containerID="b9b0f1fd23ca33969e50a5306194756a354b8d56ef2f5c7f6603633318a3a10d" Jul 12 19:31:53.625868 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:31:53.625840 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-m9vxk\" is forbidden: User \"system:node:ip-10-0-130-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-167.ec2.internal' and this object" podUID="5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m9vxk" Jul 12 19:31:53.627064 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.626999 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pmvq9" podStartSLOduration=1.707811418 podStartE2EDuration="22.626981467s" totalImagesPullingTime="20.919170049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:31:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:31:53.624052292 +0000 UTC m=+722.217326397" watchObservedRunningTime="2026-07-12 19:31:53.626981467 +0000 UTC m=+722.220255571" Jul 12 19:31:53.627271 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.627255 2564 scope.go:117] "RemoveContainer" containerID="b9b0f1fd23ca33969e50a5306194756a354b8d56ef2f5c7f6603633318a3a10d" Jul 12 19:31:53.627548 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:31:53.627531 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b0f1fd23ca33969e50a5306194756a354b8d56ef2f5c7f6603633318a3a10d\": container with ID starting with b9b0f1fd23ca33969e50a5306194756a354b8d56ef2f5c7f6603633318a3a10d not found: ID does not exist" containerID="b9b0f1fd23ca33969e50a5306194756a354b8d56ef2f5c7f6603633318a3a10d" Jul 12 19:31:53.627596 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.627555 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b0f1fd23ca33969e50a5306194756a354b8d56ef2f5c7f6603633318a3a10d"} err="failed to get container status \"b9b0f1fd23ca33969e50a5306194756a354b8d56ef2f5c7f6603633318a3a10d\": rpc error: code = NotFound desc = could not find container \"b9b0f1fd23ca33969e50a5306194756a354b8d56ef2f5c7f6603633318a3a10d\": container with ID starting with b9b0f1fd23ca33969e50a5306194756a354b8d56ef2f5c7f6603633318a3a10d not found: ID does not exist" Jul 12 19:31:53.650607 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:31:53.650567 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7cm7z\" is forbidden: User \"system:node:ip-10-0-130-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-167.ec2.internal' and this object" podUID="490ff7fd-bf14-402c-84f8-beb2ddc2ba7f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7cm7z" Jul 12 19:31:53.650740 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.650677 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z" podStartSLOduration=11.650659055 podStartE2EDuration="11.650659055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:31:53.64851242 +0000 UTC m=+722.241786530" watchObservedRunningTime="2026-07-12 19:31:53.650659055 +0000 UTC m=+722.243933156" Jul 12 19:31:53.667940 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.667896 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kxblk" podStartSLOduration=11.66788319 podStartE2EDuration="11.66788319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:31:53.667192534 +0000 UTC m=+722.260466635" watchObservedRunningTime="2026-07-12 19:31:53.66788319 +0000 UTC m=+722.261157289" Jul 12 19:31:53.917997 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.917965 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="490ff7fd-bf14-402c-84f8-beb2ddc2ba7f" path="/var/lib/kubelet/pods/490ff7fd-bf14-402c-84f8-beb2ddc2ba7f/volumes" Jul 12 19:31:53.918329 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:31:53.918314 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3" path="/var/lib/kubelet/pods/5965ad9f-2a6d-46d0-a7e8-36d837d7a7f3/volumes" Jul 12 19:32:00.669610 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:00.669573 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:00.767060 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:00.767001 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:03.140311 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:03.140264 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:03.209003 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:03.208948 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:04.620752 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:04.620720 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kxblk" Jul 12 19:32:04.621156 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:04.620900 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z" Jul 12 19:32:07.468198 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:07.468159 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:07.569620 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:07.569592 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:11.071325 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:11.071286 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:11.170979 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:11.170948 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:20.956410 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:20.956369 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2"] Jul 12 19:32:20.980099 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:20.980059 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:21.095344 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.095311 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2"] Jul 12 19:32:21.095495 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.095418 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.097898 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.097880 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vdtlz\"" Jul 12 19:32:21.202655 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.202623 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371d7bb4-bbea-4dac-8af4-67bcce7bc561-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.202655 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.202659 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371d7bb4-bbea-4dac-8af4-67bcce7bc561-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.202850 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.202679 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371d7bb4-bbea-4dac-8af4-67bcce7bc561-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.202850 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.202759 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371d7bb4-bbea-4dac-8af4-67bcce7bc561-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.202850 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.202828 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74gzv\" (UniqueName: \"kubernetes.io/projected/371d7bb4-bbea-4dac-8af4-67bcce7bc561-kube-api-access-74gzv\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.202957 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.202856 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371d7bb4-bbea-4dac-8af4-67bcce7bc561-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.202957 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.202881 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371d7bb4-bbea-4dac-8af4-67bcce7bc561-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.202957 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.202942 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371d7bb4-bbea-4dac-8af4-67bcce7bc561-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.203065 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.202969 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371d7bb4-bbea-4dac-8af4-67bcce7bc561-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.278610 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.278535 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:21.303537 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.303506 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371d7bb4-bbea-4dac-8af4-67bcce7bc561-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.303537 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.303541 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371d7bb4-bbea-4dac-8af4-67bcce7bc561-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.303775 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.303570 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371d7bb4-bbea-4dac-8af4-67bcce7bc561-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.303775 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.303602 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371d7bb4-bbea-4dac-8af4-67bcce7bc561-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.303775 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.303627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74gzv\" (UniqueName: \"kubernetes.io/projected/371d7bb4-bbea-4dac-8af4-67bcce7bc561-kube-api-access-74gzv\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.303775 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.303656 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371d7bb4-bbea-4dac-8af4-67bcce7bc561-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.303775 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.303676 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371d7bb4-bbea-4dac-8af4-67bcce7bc561-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.303775 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.303709 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371d7bb4-bbea-4dac-8af4-67bcce7bc561-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.303775 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.303748 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371d7bb4-bbea-4dac-8af4-67bcce7bc561-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.304252 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.303969 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371d7bb4-bbea-4dac-8af4-67bcce7bc561-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.304252 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.304000 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371d7bb4-bbea-4dac-8af4-67bcce7bc561-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.304252 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.304176 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371d7bb4-bbea-4dac-8af4-67bcce7bc561-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.304252 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.304182 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371d7bb4-bbea-4dac-8af4-67bcce7bc561-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.304807 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.304785 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371d7bb4-bbea-4dac-8af4-67bcce7bc561-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.306138 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.306122 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371d7bb4-bbea-4dac-8af4-67bcce7bc561-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.306273 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.306249 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371d7bb4-bbea-4dac-8af4-67bcce7bc561-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.314898 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.314868 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74gzv\" (UniqueName: \"kubernetes.io/projected/371d7bb4-bbea-4dac-8af4-67bcce7bc561-kube-api-access-74gzv\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.315045 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.314950 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371d7bb4-bbea-4dac-8af4-67bcce7bc561-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-lb5c2\" (UID: \"371d7bb4-bbea-4dac-8af4-67bcce7bc561\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.378858 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.378821 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:21.405784 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.405748 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:21.580588 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.580560 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2"] Jul 12 19:32:21.581949 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.581890 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:21.582993 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:32:21.582962 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371d7bb4_bbea_4dac_8af4_67bcce7bc561.slice/crio-a6c1f3bf6869300e36a04311353d326712a0fbba723ee6b1ff593d2432dc5457 WatchSource:0}: Error finding container a6c1f3bf6869300e36a04311353d326712a0fbba723ee6b1ff593d2432dc5457: Status 404 returned error can't find the container with id a6c1f3bf6869300e36a04311353d326712a0fbba723ee6b1ff593d2432dc5457 Jul 12 19:32:21.584777 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.584749 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:32:21.584861 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.584808 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:32:21.584861 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.584833 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:32:21.754701 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.754665 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" event={"ID":"371d7bb4-bbea-4dac-8af4-67bcce7bc561","Type":"ContainerStarted","Data":"02e4d42f0768191749b6a3b439afd83044d261b7da31d6584e95a508eb9a1376"} Jul 12 19:32:21.754880 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.754731 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" event={"ID":"371d7bb4-bbea-4dac-8af4-67bcce7bc561","Type":"ContainerStarted","Data":"a6c1f3bf6869300e36a04311353d326712a0fbba723ee6b1ff593d2432dc5457"} Jul 12 19:32:21.776095 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:21.776039 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" podStartSLOduration=1.7760072839999999 podStartE2EDuration="1.776007284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:32:21.77450078 +0000 UTC m=+750.367774880" watchObservedRunningTime="2026-07-12 19:32:21.776007284 +0000 UTC m=+750.369281452" Jul 12 19:32:22.406697 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:22.406657 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:22.411468 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:22.411443 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:22.759661 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:22.759582 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:22.760546 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:22.760515 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:32:23.717796 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:23.717760 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:23.769270 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:23.769235 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:26.011390 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:26.011359 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:26.119661 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:26.119593 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:26.175907 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:26.175874 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:26.818763 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:26.818720 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:26.870973 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:26.870939 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:26.983478 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:26.983443 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-jx8ts"] Jul 12 19:32:27.019181 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:27.019149 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jx8ts"] Jul 12 19:32:27.019546 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:27.019228 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jx8ts" Jul 12 19:32:27.021666 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:27.021649 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jdrdk\"" Jul 12 19:32:27.159721 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:27.159685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf5vb\" (UniqueName: \"kubernetes.io/projected/463bcb16-4694-4f08-813c-9411038222dc-kube-api-access-pf5vb\") pod \"authorino-75d556bd77-jx8ts\" (UID: \"463bcb16-4694-4f08-813c-9411038222dc\") " pod="kuadrant-system/authorino-75d556bd77-jx8ts" Jul 12 19:32:27.261015 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:27.260981 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf5vb\" (UniqueName: \"kubernetes.io/projected/463bcb16-4694-4f08-813c-9411038222dc-kube-api-access-pf5vb\") pod \"authorino-75d556bd77-jx8ts\" (UID: \"463bcb16-4694-4f08-813c-9411038222dc\") " pod="kuadrant-system/authorino-75d556bd77-jx8ts" Jul 12 19:32:27.268809 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:27.268787 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf5vb\" (UniqueName: \"kubernetes.io/projected/463bcb16-4694-4f08-813c-9411038222dc-kube-api-access-pf5vb\") pod \"authorino-75d556bd77-jx8ts\" (UID: \"463bcb16-4694-4f08-813c-9411038222dc\") " pod="kuadrant-system/authorino-75d556bd77-jx8ts" Jul 12 19:32:27.329157 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:27.329131 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jx8ts" Jul 12 19:32:27.462685 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:27.462644 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jx8ts"] Jul 12 19:32:27.470820 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:32:27.470792 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod463bcb16_4694_4f08_813c_9411038222dc.slice/crio-bdcc12d2eb7f12ca1b37a282726b7f91a9d5d00837b6c0bd32d891a6acf57f8f WatchSource:0}: Error finding container bdcc12d2eb7f12ca1b37a282726b7f91a9d5d00837b6c0bd32d891a6acf57f8f: Status 404 returned error can't find the container with id bdcc12d2eb7f12ca1b37a282726b7f91a9d5d00837b6c0bd32d891a6acf57f8f Jul 12 19:32:27.783741 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:27.783656 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jx8ts" event={"ID":"463bcb16-4694-4f08-813c-9411038222dc","Type":"ContainerStarted","Data":"bdcc12d2eb7f12ca1b37a282726b7f91a9d5d00837b6c0bd32d891a6acf57f8f"} Jul 12 19:32:30.802479 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:30.802434 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jx8ts" event={"ID":"463bcb16-4694-4f08-813c-9411038222dc","Type":"ContainerStarted","Data":"2245f5844b1ea7a0301b86c4d45ee11167245bd89c4122a8122b007a3eb95eac"} Jul 12 19:32:30.816753 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:30.816705 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-jx8ts" podStartSLOduration=1.943609457 podStartE2EDuration="4.816691516s" totalImagesPullingTime="2.873082059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:32:30.816040051 +0000 UTC m=+759.409314141" watchObservedRunningTime="2026-07-12 19:32:30.816691516 +0000 UTC m=+759.409965614" Jul 12 19:32:43.963420 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:43.963380 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:44.062989 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:44.062954 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:44.469769 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:44.469730 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:56.491532 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:56.491494 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:56.670083 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:56.670038 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:58.222169 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:58.222137 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jx8ts"] Jul 12 19:32:58.222677 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:58.222308 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-jx8ts" podUID="463bcb16-4694-4f08-813c-9411038222dc" containerName="authorino" containerID="cri-o://2245f5844b1ea7a0301b86c4d45ee11167245bd89c4122a8122b007a3eb95eac" gracePeriod=30 Jul 12 19:32:58.476621 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:58.476566 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jx8ts" Jul 12 19:32:58.513077 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:58.513048 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf5vb\" (UniqueName: \"kubernetes.io/projected/463bcb16-4694-4f08-813c-9411038222dc-kube-api-access-pf5vb\") pod \"463bcb16-4694-4f08-813c-9411038222dc\" (UID: \"463bcb16-4694-4f08-813c-9411038222dc\") " Jul 12 19:32:58.515229 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:58.515198 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463bcb16-4694-4f08-813c-9411038222dc-kube-api-access-pf5vb" (OuterVolumeSpecName: "kube-api-access-pf5vb") pod "463bcb16-4694-4f08-813c-9411038222dc" (UID: "463bcb16-4694-4f08-813c-9411038222dc"). InnerVolumeSpecName "kube-api-access-pf5vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:32:58.614504 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:58.614474 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf5vb\" (UniqueName: \"kubernetes.io/projected/463bcb16-4694-4f08-813c-9411038222dc-kube-api-access-pf5vb\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:32:58.938383 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:58.938350 2564 generic.go:356] "Generic (PLEG): container finished" podID="463bcb16-4694-4f08-813c-9411038222dc" containerID="2245f5844b1ea7a0301b86c4d45ee11167245bd89c4122a8122b007a3eb95eac" exitCode=0 Jul 12 19:32:58.938767 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:58.938744 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jx8ts" event={"ID":"463bcb16-4694-4f08-813c-9411038222dc","Type":"ContainerDied","Data":"2245f5844b1ea7a0301b86c4d45ee11167245bd89c4122a8122b007a3eb95eac"} Jul 12 19:32:58.938977 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:58.938943 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jx8ts" event={"ID":"463bcb16-4694-4f08-813c-9411038222dc","Type":"ContainerDied","Data":"bdcc12d2eb7f12ca1b37a282726b7f91a9d5d00837b6c0bd32d891a6acf57f8f"} Jul 12 19:32:58.939155 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:58.939141 2564 scope.go:117] "RemoveContainer" containerID="2245f5844b1ea7a0301b86c4d45ee11167245bd89c4122a8122b007a3eb95eac" Jul 12 19:32:58.939499 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:58.939470 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jx8ts" Jul 12 19:32:58.948636 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:58.948619 2564 scope.go:117] "RemoveContainer" containerID="2245f5844b1ea7a0301b86c4d45ee11167245bd89c4122a8122b007a3eb95eac" Jul 12 19:32:58.948895 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:32:58.948876 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2245f5844b1ea7a0301b86c4d45ee11167245bd89c4122a8122b007a3eb95eac\": container with ID starting with 2245f5844b1ea7a0301b86c4d45ee11167245bd89c4122a8122b007a3eb95eac not found: ID does not exist" containerID="2245f5844b1ea7a0301b86c4d45ee11167245bd89c4122a8122b007a3eb95eac" Jul 12 19:32:58.948942 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:58.948904 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2245f5844b1ea7a0301b86c4d45ee11167245bd89c4122a8122b007a3eb95eac"} err="failed to get container status \"2245f5844b1ea7a0301b86c4d45ee11167245bd89c4122a8122b007a3eb95eac\": rpc error: code = NotFound desc = could not find container \"2245f5844b1ea7a0301b86c4d45ee11167245bd89c4122a8122b007a3eb95eac\": container with ID starting with 2245f5844b1ea7a0301b86c4d45ee11167245bd89c4122a8122b007a3eb95eac not found: ID does not exist" Jul 12 19:32:58.971078 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:58.971054 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jx8ts"] Jul 12 19:32:58.994512 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:58.994489 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jx8ts"] Jul 12 19:32:59.920393 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:32:59.920361 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="463bcb16-4694-4f08-813c-9411038222dc" path="/var/lib/kubelet/pods/463bcb16-4694-4f08-813c-9411038222dc/volumes" Jul 12 19:33:02.625300 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:02.625259 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:33:02.725987 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:02.725949 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:33:02.805504 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:02.805472 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:33:02.856491 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:02.856458 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:33:06.276731 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:06.276683 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:33:06.368677 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:06.368633 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:33:37.776327 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:37.776288 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:33:38.188610 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:38.188578 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:33:38.621444 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:38.621361 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:33:38.696964 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:38.696924 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:33:38.735643 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:38.735611 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-2hh49"] Jul 12 19:33:38.736012 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:38.735934 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="463bcb16-4694-4f08-813c-9411038222dc" containerName="authorino" Jul 12 19:33:38.736012 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:38.735945 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="463bcb16-4694-4f08-813c-9411038222dc" containerName="authorino" Jul 12 19:33:38.736012 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:38.735995 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="463bcb16-4694-4f08-813c-9411038222dc" containerName="authorino" Jul 12 19:33:38.739253 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:38.739233 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-2hh49" Jul 12 19:33:38.741804 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:38.741775 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 12 19:33:38.741906 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:38.741780 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bs5nc\"" Jul 12 19:33:38.749964 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:38.749939 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-2hh49"] Jul 12 19:33:38.781780 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:38.781745 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:33:38.844700 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:38.844667 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxk6l\" (UniqueName: \"kubernetes.io/projected/c071e130-cd90-4008-8e90-25090615fb17-kube-api-access-lxk6l\") pod \"payload-processing-58f86c7798-2hh49\" (UID: \"c071e130-cd90-4008-8e90-25090615fb17\") " pod="openshift-ingress/payload-processing-58f86c7798-2hh49" Jul 12 19:33:38.844862 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:38.844709 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c071e130-cd90-4008-8e90-25090615fb17-plugins-config-volume\") pod \"payload-processing-58f86c7798-2hh49\" (UID: \"c071e130-cd90-4008-8e90-25090615fb17\") " pod="openshift-ingress/payload-processing-58f86c7798-2hh49" Jul 12 19:33:38.945550 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:38.945515 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxk6l\" (UniqueName: \"kubernetes.io/projected/c071e130-cd90-4008-8e90-25090615fb17-kube-api-access-lxk6l\") pod \"payload-processing-58f86c7798-2hh49\" (UID: \"c071e130-cd90-4008-8e90-25090615fb17\") " pod="openshift-ingress/payload-processing-58f86c7798-2hh49" Jul 12 19:33:38.945738 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:38.945568 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c071e130-cd90-4008-8e90-25090615fb17-plugins-config-volume\") pod \"payload-processing-58f86c7798-2hh49\" (UID: \"c071e130-cd90-4008-8e90-25090615fb17\") " pod="openshift-ingress/payload-processing-58f86c7798-2hh49" Jul 12 19:33:38.946298 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:38.946273 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c071e130-cd90-4008-8e90-25090615fb17-plugins-config-volume\") pod \"payload-processing-58f86c7798-2hh49\" (UID: \"c071e130-cd90-4008-8e90-25090615fb17\") " pod="openshift-ingress/payload-processing-58f86c7798-2hh49" Jul 12 19:33:38.953289 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:38.953265 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxk6l\" (UniqueName: \"kubernetes.io/projected/c071e130-cd90-4008-8e90-25090615fb17-kube-api-access-lxk6l\") pod \"payload-processing-58f86c7798-2hh49\" (UID: \"c071e130-cd90-4008-8e90-25090615fb17\") " pod="openshift-ingress/payload-processing-58f86c7798-2hh49" Jul 12 19:33:39.052564 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:39.052530 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-2hh49" Jul 12 19:33:39.187445 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:39.187423 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-2hh49"] Jul 12 19:33:39.189292 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:33:39.189266 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc071e130_cd90_4008_8e90_25090615fb17.slice/crio-6d2fe03be73e42d0dfeb31fe10d87bc66e5746b807dd95a8a2cbc1e60e049832 WatchSource:0}: Error finding container 6d2fe03be73e42d0dfeb31fe10d87bc66e5746b807dd95a8a2cbc1e60e049832: Status 404 returned error can't find the container with id 6d2fe03be73e42d0dfeb31fe10d87bc66e5746b807dd95a8a2cbc1e60e049832 Jul 12 19:33:40.142519 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:40.142478 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-2hh49" event={"ID":"c071e130-cd90-4008-8e90-25090615fb17","Type":"ContainerStarted","Data":"6d2fe03be73e42d0dfeb31fe10d87bc66e5746b807dd95a8a2cbc1e60e049832"} Jul 12 19:33:41.149512 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:41.149479 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-2hh49" event={"ID":"c071e130-cd90-4008-8e90-25090615fb17","Type":"ContainerStarted","Data":"01ca43940cae0a509fbc49eea937390f023024a90bdd47a0c789198ddbe39178"} Jul 12 19:33:41.149968 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:41.149647 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-2hh49" Jul 12 19:33:41.165944 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:41.165899 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-2hh49" podStartSLOduration=1.717461533 podStartE2EDuration="3.165879512s" totalImagesPullingTime="1.448417979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:33:41.164887907 +0000 UTC m=+829.758162004" watchObservedRunningTime="2026-07-12 19:33:41.165879512 +0000 UTC m=+829.759153611" Jul 12 19:33:47.184340 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:47.184307 2564 generic.go:356] "Generic (PLEG): container finished" podID="371d7bb4-bbea-4dac-8af4-67bcce7bc561" containerID="02e4d42f0768191749b6a3b439afd83044d261b7da31d6584e95a508eb9a1376" exitCode=0 Jul 12 19:33:47.184746 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:47.184383 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" event={"ID":"371d7bb4-bbea-4dac-8af4-67bcce7bc561","Type":"ContainerDied","Data":"02e4d42f0768191749b6a3b439afd83044d261b7da31d6584e95a508eb9a1376"} Jul 12 19:33:47.184788 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:47.184762 2564 scope.go:117] "RemoveContainer" containerID="02e4d42f0768191749b6a3b439afd83044d261b7da31d6584e95a508eb9a1376" Jul 12 19:33:47.185707 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:47.185683 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:33:47.185762 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:47.185749 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:33:47.185798 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:47.185785 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:33:48.191188 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:48.191157 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" event={"ID":"371d7bb4-bbea-4dac-8af4-67bcce7bc561","Type":"ContainerStarted","Data":"8607ef09725e0ea65b747356bc06574e3918be7d322515e3072b9ae1d9861ed0"} Jul 12 19:33:48.406836 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:48.406797 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:33:48.407792 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:48.407766 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-lb5c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 12 19:33:48.407922 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:48.407815 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" podUID="371d7bb4-bbea-4dac-8af4-67bcce7bc561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 12 19:33:48.899290 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:48.899254 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:33:48.999168 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:48.999132 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:33:49.406518 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:49.406483 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-lb5c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 12 19:33:49.406889 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:49.406548 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" podUID="371d7bb4-bbea-4dac-8af4-67bcce7bc561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 12 19:33:50.406939 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:50.406896 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-lb5c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 12 19:33:50.407337 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:50.406961 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" podUID="371d7bb4-bbea-4dac-8af4-67bcce7bc561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 12 19:33:51.406368 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:51.406333 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-lb5c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 12 19:33:51.406534 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:51.406386 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" podUID="371d7bb4-bbea-4dac-8af4-67bcce7bc561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 12 19:33:52.159816 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:52.159779 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-2hh49" Jul 12 19:33:52.410109 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:52.410033 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:33:52.410358 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:52.410319 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:33:52.410905 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:33:52.410891 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lb5c2" Jul 12 19:34:07.307679 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:07.307633 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:34:18.994463 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:18.994427 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:34:28.004658 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:28.004623 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:34:29.363086 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.363011 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-r44np"] Jul 12 19:34:29.366726 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.366704 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:29.370649 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.370585 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-c45j6\"" Jul 12 19:34:29.370649 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.370609 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 12 19:34:29.370863 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.370786 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 12 19:34:29.370951 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.370932 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 12 19:34:29.378185 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.378163 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-r44np"] Jul 12 19:34:29.395657 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.395618 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:34:29.454973 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.454936 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c602a26e-93f6-4960-9d58-4fc728139e8a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r44np\" (UID: \"c602a26e-93f6-4960-9d58-4fc728139e8a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:29.455143 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.454983 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c602a26e-93f6-4960-9d58-4fc728139e8a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r44np\" (UID: \"c602a26e-93f6-4960-9d58-4fc728139e8a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:29.455143 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.455000 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6qfr\" (UniqueName: \"kubernetes.io/projected/c602a26e-93f6-4960-9d58-4fc728139e8a-kube-api-access-c6qfr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r44np\" (UID: \"c602a26e-93f6-4960-9d58-4fc728139e8a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:29.455143 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.455071 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c602a26e-93f6-4960-9d58-4fc728139e8a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r44np\" (UID: \"c602a26e-93f6-4960-9d58-4fc728139e8a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:29.455143 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.455113 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c602a26e-93f6-4960-9d58-4fc728139e8a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r44np\" (UID: \"c602a26e-93f6-4960-9d58-4fc728139e8a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:29.455143 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.455137 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c602a26e-93f6-4960-9d58-4fc728139e8a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r44np\" (UID: \"c602a26e-93f6-4960-9d58-4fc728139e8a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:29.555788 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.555748 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c602a26e-93f6-4960-9d58-4fc728139e8a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r44np\" (UID: \"c602a26e-93f6-4960-9d58-4fc728139e8a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:29.555965 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.555796 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c602a26e-93f6-4960-9d58-4fc728139e8a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r44np\" (UID: \"c602a26e-93f6-4960-9d58-4fc728139e8a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:29.555965 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.555813 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6qfr\" (UniqueName: \"kubernetes.io/projected/c602a26e-93f6-4960-9d58-4fc728139e8a-kube-api-access-c6qfr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r44np\" (UID: \"c602a26e-93f6-4960-9d58-4fc728139e8a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:29.555965 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.555854 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c602a26e-93f6-4960-9d58-4fc728139e8a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r44np\" (UID: \"c602a26e-93f6-4960-9d58-4fc728139e8a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:29.555965 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.555878 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c602a26e-93f6-4960-9d58-4fc728139e8a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r44np\" (UID: \"c602a26e-93f6-4960-9d58-4fc728139e8a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:29.555965 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.555893 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c602a26e-93f6-4960-9d58-4fc728139e8a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r44np\" (UID: \"c602a26e-93f6-4960-9d58-4fc728139e8a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:29.556262 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.556145 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c602a26e-93f6-4960-9d58-4fc728139e8a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r44np\" (UID: \"c602a26e-93f6-4960-9d58-4fc728139e8a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:29.556311 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.556258 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c602a26e-93f6-4960-9d58-4fc728139e8a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r44np\" (UID: \"c602a26e-93f6-4960-9d58-4fc728139e8a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:29.556345 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.556313 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c602a26e-93f6-4960-9d58-4fc728139e8a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r44np\" (UID: \"c602a26e-93f6-4960-9d58-4fc728139e8a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:29.558143 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.558121 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c602a26e-93f6-4960-9d58-4fc728139e8a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r44np\" (UID: \"c602a26e-93f6-4960-9d58-4fc728139e8a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:29.558399 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.558381 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c602a26e-93f6-4960-9d58-4fc728139e8a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r44np\" (UID: \"c602a26e-93f6-4960-9d58-4fc728139e8a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:29.564225 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.564208 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6qfr\" (UniqueName: \"kubernetes.io/projected/c602a26e-93f6-4960-9d58-4fc728139e8a-kube-api-access-c6qfr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r44np\" (UID: \"c602a26e-93f6-4960-9d58-4fc728139e8a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:29.679219 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.679182 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:29.814469 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:29.814438 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-r44np"] Jul 12 19:34:29.816296 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:34:29.816247 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc602a26e_93f6_4960_9d58_4fc728139e8a.slice/crio-230ed8a58c114d839ea1559992329eb0295c871819cbf0ba4eb6dbddca4db673 WatchSource:0}: Error finding container 230ed8a58c114d839ea1559992329eb0295c871819cbf0ba4eb6dbddca4db673: Status 404 returned error can't find the container with id 230ed8a58c114d839ea1559992329eb0295c871819cbf0ba4eb6dbddca4db673 Jul 12 19:34:30.401095 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:30.401055 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" event={"ID":"c602a26e-93f6-4960-9d58-4fc728139e8a","Type":"ContainerStarted","Data":"230ed8a58c114d839ea1559992329eb0295c871819cbf0ba4eb6dbddca4db673"} Jul 12 19:34:31.407863 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:31.407831 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" event={"ID":"c602a26e-93f6-4960-9d58-4fc728139e8a","Type":"ContainerStarted","Data":"759ec52ccf446527fe9f383b0e469448dda2bc0969251821358fbcfb35fb3db6"} Jul 12 19:34:31.425877 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:31.425823 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" podStartSLOduration=1.364414279 podStartE2EDuration="2.425807561s" totalImagesPullingTime="1.061393282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:34:31.424701359 +0000 UTC m=+880.017975481" watchObservedRunningTime="2026-07-12 19:34:31.425807561 +0000 UTC m=+880.019081659" Jul 12 19:34:33.380763 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:33.380732 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:34:39.679545 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:39.679511 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:39.679938 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:39.679579 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:39.687293 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:39.687249 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:34:39.692228 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:39.692202 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:34:40.466843 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:34:40.466815 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r44np" Jul 12 19:35:00.565721 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.565638 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc"] Jul 12 19:35:00.570263 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.570242 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:00.572680 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.572661 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 12 19:35:00.577853 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.577830 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc"] Jul 12 19:35:00.592259 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.592219 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:35:00.615440 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.615417 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b4fad4c-ed4c-4470-9f13-663f6de6d7da-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc\" (UID: \"6b4fad4c-ed4c-4470-9f13-663f6de6d7da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:00.615555 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.615450 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b4fad4c-ed4c-4470-9f13-663f6de6d7da-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc\" (UID: \"6b4fad4c-ed4c-4470-9f13-663f6de6d7da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:00.615555 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.615489 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b4fad4c-ed4c-4470-9f13-663f6de6d7da-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc\" (UID: \"6b4fad4c-ed4c-4470-9f13-663f6de6d7da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:00.615555 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.615536 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp2hb\" (UniqueName: \"kubernetes.io/projected/6b4fad4c-ed4c-4470-9f13-663f6de6d7da-kube-api-access-kp2hb\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc\" (UID: \"6b4fad4c-ed4c-4470-9f13-663f6de6d7da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:00.615659 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.615568 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b4fad4c-ed4c-4470-9f13-663f6de6d7da-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc\" (UID: \"6b4fad4c-ed4c-4470-9f13-663f6de6d7da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:00.615659 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.615596 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b4fad4c-ed4c-4470-9f13-663f6de6d7da-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc\" (UID: \"6b4fad4c-ed4c-4470-9f13-663f6de6d7da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:00.716192 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.716148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b4fad4c-ed4c-4470-9f13-663f6de6d7da-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc\" (UID: \"6b4fad4c-ed4c-4470-9f13-663f6de6d7da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:00.716345 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.716197 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b4fad4c-ed4c-4470-9f13-663f6de6d7da-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc\" (UID: \"6b4fad4c-ed4c-4470-9f13-663f6de6d7da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:00.716345 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.716247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b4fad4c-ed4c-4470-9f13-663f6de6d7da-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc\" (UID: \"6b4fad4c-ed4c-4470-9f13-663f6de6d7da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:00.716345 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.716270 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp2hb\" (UniqueName: \"kubernetes.io/projected/6b4fad4c-ed4c-4470-9f13-663f6de6d7da-kube-api-access-kp2hb\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc\" (UID: \"6b4fad4c-ed4c-4470-9f13-663f6de6d7da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:00.716345 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.716302 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b4fad4c-ed4c-4470-9f13-663f6de6d7da-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc\" (UID: \"6b4fad4c-ed4c-4470-9f13-663f6de6d7da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:00.716345 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.716332 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b4fad4c-ed4c-4470-9f13-663f6de6d7da-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc\" (UID: \"6b4fad4c-ed4c-4470-9f13-663f6de6d7da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:00.716625 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.716608 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b4fad4c-ed4c-4470-9f13-663f6de6d7da-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc\" (UID: \"6b4fad4c-ed4c-4470-9f13-663f6de6d7da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:00.716717 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.716697 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b4fad4c-ed4c-4470-9f13-663f6de6d7da-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc\" (UID: \"6b4fad4c-ed4c-4470-9f13-663f6de6d7da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:00.716883 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.716863 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b4fad4c-ed4c-4470-9f13-663f6de6d7da-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc\" (UID: \"6b4fad4c-ed4c-4470-9f13-663f6de6d7da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:00.718986 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.718946 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b4fad4c-ed4c-4470-9f13-663f6de6d7da-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc\" (UID: \"6b4fad4c-ed4c-4470-9f13-663f6de6d7da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:00.719270 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.719249 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b4fad4c-ed4c-4470-9f13-663f6de6d7da-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc\" (UID: \"6b4fad4c-ed4c-4470-9f13-663f6de6d7da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:00.723651 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.723623 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp2hb\" (UniqueName: \"kubernetes.io/projected/6b4fad4c-ed4c-4470-9f13-663f6de6d7da-kube-api-access-kp2hb\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc\" (UID: \"6b4fad4c-ed4c-4470-9f13-663f6de6d7da\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:00.880632 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:00.880538 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:01.023373 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:01.023350 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc"] Jul 12 19:35:01.025382 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:35:01.025348 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b4fad4c_ed4c_4470_9f13_663f6de6d7da.slice/crio-f846d8eead29888ea483e0cbb31e0698f54ecc1ec67725956e04fdc41b54641b WatchSource:0}: Error finding container f846d8eead29888ea483e0cbb31e0698f54ecc1ec67725956e04fdc41b54641b: Status 404 returned error can't find the container with id f846d8eead29888ea483e0cbb31e0698f54ecc1ec67725956e04fdc41b54641b Jul 12 19:35:01.559042 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:01.558943 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" event={"ID":"6b4fad4c-ed4c-4470-9f13-663f6de6d7da","Type":"ContainerStarted","Data":"888baf3d96ddfd4c4679691fa23e70d5b621cfab0678f83b9582b394cf3ab7b3"} Jul 12 19:35:01.559042 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:01.558986 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" event={"ID":"6b4fad4c-ed4c-4470-9f13-663f6de6d7da","Type":"ContainerStarted","Data":"f846d8eead29888ea483e0cbb31e0698f54ecc1ec67725956e04fdc41b54641b"} Jul 12 19:35:01.577008 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:01.576960 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" podStartSLOduration=1.409168247 podStartE2EDuration="1.576946201s" totalImagesPullingTime="167.777954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:35:01.574914639 +0000 UTC m=+910.168188738" watchObservedRunningTime="2026-07-12 19:35:01.576946201 +0000 UTC m=+910.170220298" Jul 12 19:35:10.881599 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:10.881556 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:10.881985 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:10.881621 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:10.894053 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:10.894004 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:35:11.622207 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:35:11.622177 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc" Jul 12 19:36:07.206310 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:07.206274 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68b74f5cfd-jcnhj"] Jul 12 19:36:07.209504 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:07.209483 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68b74f5cfd-jcnhj" Jul 12 19:36:07.212816 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:07.212795 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jdrdk\"" Jul 12 19:36:07.212816 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:07.212802 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 12 19:36:07.216674 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:07.216647 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68b74f5cfd-jcnhj"] Jul 12 19:36:07.276117 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:07.276087 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cfzc\" (UniqueName: \"kubernetes.io/projected/89f785c3-9ed2-49fa-9cae-9447fa673963-kube-api-access-7cfzc\") pod \"authorino-68b74f5cfd-jcnhj\" (UID: \"89f785c3-9ed2-49fa-9cae-9447fa673963\") " pod="kuadrant-system/authorino-68b74f5cfd-jcnhj" Jul 12 19:36:07.276245 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:07.276124 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/89f785c3-9ed2-49fa-9cae-9447fa673963-tls-cert\") pod \"authorino-68b74f5cfd-jcnhj\" (UID: \"89f785c3-9ed2-49fa-9cae-9447fa673963\") " pod="kuadrant-system/authorino-68b74f5cfd-jcnhj" Jul 12 19:36:07.377013 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:07.376980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cfzc\" (UniqueName: \"kubernetes.io/projected/89f785c3-9ed2-49fa-9cae-9447fa673963-kube-api-access-7cfzc\") pod \"authorino-68b74f5cfd-jcnhj\" (UID: \"89f785c3-9ed2-49fa-9cae-9447fa673963\") " pod="kuadrant-system/authorino-68b74f5cfd-jcnhj" Jul 12 19:36:07.377197 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:07.377046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/89f785c3-9ed2-49fa-9cae-9447fa673963-tls-cert\") pod \"authorino-68b74f5cfd-jcnhj\" (UID: \"89f785c3-9ed2-49fa-9cae-9447fa673963\") " pod="kuadrant-system/authorino-68b74f5cfd-jcnhj" Jul 12 19:36:07.379688 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:07.379658 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/89f785c3-9ed2-49fa-9cae-9447fa673963-tls-cert\") pod \"authorino-68b74f5cfd-jcnhj\" (UID: \"89f785c3-9ed2-49fa-9cae-9447fa673963\") " pod="kuadrant-system/authorino-68b74f5cfd-jcnhj" Jul 12 19:36:07.384373 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:07.384350 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cfzc\" (UniqueName: \"kubernetes.io/projected/89f785c3-9ed2-49fa-9cae-9447fa673963-kube-api-access-7cfzc\") pod \"authorino-68b74f5cfd-jcnhj\" (UID: \"89f785c3-9ed2-49fa-9cae-9447fa673963\") " pod="kuadrant-system/authorino-68b74f5cfd-jcnhj" Jul 12 19:36:07.520104 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:07.520028 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68b74f5cfd-jcnhj" Jul 12 19:36:07.536256 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:07.536223 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:36:07.648850 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:07.648819 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:36:07.670159 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:07.670132 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68b74f5cfd-jcnhj"] Jul 12 19:36:07.673268 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:36:07.673241 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89f785c3_9ed2_49fa_9cae_9447fa673963.slice/crio-c8e6d9d15b0d58b6f055c6c48efecb0378464abab888fa9c9765fde25402c2ad WatchSource:0}: Error finding container c8e6d9d15b0d58b6f055c6c48efecb0378464abab888fa9c9765fde25402c2ad: Status 404 returned error can't find the container with id c8e6d9d15b0d58b6f055c6c48efecb0378464abab888fa9c9765fde25402c2ad Jul 12 19:36:07.886550 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:07.886469 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68b74f5cfd-jcnhj" event={"ID":"89f785c3-9ed2-49fa-9cae-9447fa673963","Type":"ContainerStarted","Data":"c8e6d9d15b0d58b6f055c6c48efecb0378464abab888fa9c9765fde25402c2ad"} Jul 12 19:36:08.654494 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:08.654441 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:36:08.745613 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:08.745572 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:36:08.895125 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:08.895081 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68b74f5cfd-jcnhj" event={"ID":"89f785c3-9ed2-49fa-9cae-9447fa673963","Type":"ContainerStarted","Data":"98d79ad2918ce648ed5f2bb7bda8fec526c52a676abe2b2fb93d81220707b4c7"} Jul 12 19:36:08.911133 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:08.911009 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-68b74f5cfd-jcnhj" podStartSLOduration=1.47061289 podStartE2EDuration="1.910992259s" totalImagesPullingTime="440.379369ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:36:08.909684812 +0000 UTC m=+977.502958911" watchObservedRunningTime="2026-07-12 19:36:08.910992259 +0000 UTC m=+977.504266356" Jul 12 19:36:18.406528 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:18.406424 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:36:18.508301 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:36:18.508267 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:37:18.186904 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:37:18.186856 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:37:20.101816 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:37:20.101775 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:37:33.537767 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:37:33.537721 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:37:33.592840 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:37:33.592806 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:37:36.619062 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:37:36.619002 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:37:58.610080 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:37:58.609971 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:37:58.708781 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:37:58.708744 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:37:58.822984 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:37:58.822935 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:38:04.696662 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:38:04.696623 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:38:04.796976 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:38:04.796939 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:38:04.898171 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:38:04.898140 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:38:05.027749 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:38:05.027651 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:38:10.533958 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:38:10.533922 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:38:10.634256 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:38:10.634218 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:38:11.056977 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:38:11.056930 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:39:07.857502 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:39:07.857460 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:42:01.842417 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:42:01.842383 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z"] Jul 12 19:42:01.842980 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:42:01.842589 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z" podUID="e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4" containerName="manager" containerID="cri-o://5076d6ffe93c4942d3fe2fe03366a291e257dfb754abfb88f198f886f50507ac" gracePeriod=10 Jul 12 19:42:04.993344 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:42:04.993318 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z" Jul 12 19:42:05.178482 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:42:05.178453 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qz8bv\" (UniqueName: \"kubernetes.io/projected/e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4-kube-api-access-qz8bv\") pod \"e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4\" (UID: \"e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4\") " Jul 12 19:42:05.178650 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:42:05.178528 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4-extensions-socket-volume\") pod \"e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4\" (UID: \"e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4\") " Jul 12 19:42:05.178884 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:42:05.178863 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4" (UID: "e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:42:05.180656 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:42:05.180639 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4-kube-api-access-qz8bv" (OuterVolumeSpecName: "kube-api-access-qz8bv") pod "e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4" (UID: "e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4"). InnerVolumeSpecName "kube-api-access-qz8bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:42:05.279234 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:42:05.279196 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4-extensions-socket-volume\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:42:05.279234 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:42:05.279227 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz8bv\" (UniqueName: \"kubernetes.io/projected/e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4-kube-api-access-qz8bv\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:42:05.702679 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:42:05.702640 2564 generic.go:356] "Generic (PLEG): container finished" podID="e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4" containerID="5076d6ffe93c4942d3fe2fe03366a291e257dfb754abfb88f198f886f50507ac" exitCode=0 Jul 12 19:42:05.702880 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:42:05.702687 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z" Jul 12 19:42:05.702880 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:42:05.702728 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z" event={"ID":"e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4","Type":"ContainerDied","Data":"5076d6ffe93c4942d3fe2fe03366a291e257dfb754abfb88f198f886f50507ac"} Jul 12 19:42:05.702880 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:42:05.702788 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z" event={"ID":"e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4","Type":"ContainerDied","Data":"0ccfabe52e431b2107d50558a911760cb24eff670ff9348953e3eb582c9eaf93"} Jul 12 19:42:05.702880 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:42:05.702816 2564 scope.go:117] "RemoveContainer" containerID="5076d6ffe93c4942d3fe2fe03366a291e257dfb754abfb88f198f886f50507ac" Jul 12 19:42:05.712443 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:42:05.712427 2564 scope.go:117] "RemoveContainer" containerID="5076d6ffe93c4942d3fe2fe03366a291e257dfb754abfb88f198f886f50507ac" Jul 12 19:42:05.712721 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:42:05.712701 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5076d6ffe93c4942d3fe2fe03366a291e257dfb754abfb88f198f886f50507ac\": container with ID starting with 5076d6ffe93c4942d3fe2fe03366a291e257dfb754abfb88f198f886f50507ac not found: ID does not exist" containerID="5076d6ffe93c4942d3fe2fe03366a291e257dfb754abfb88f198f886f50507ac" Jul 12 19:42:05.712779 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:42:05.712729 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5076d6ffe93c4942d3fe2fe03366a291e257dfb754abfb88f198f886f50507ac"} err="failed to get container status \"5076d6ffe93c4942d3fe2fe03366a291e257dfb754abfb88f198f886f50507ac\": rpc error: code = NotFound desc = could not find container \"5076d6ffe93c4942d3fe2fe03366a291e257dfb754abfb88f198f886f50507ac\": container with ID starting with 5076d6ffe93c4942d3fe2fe03366a291e257dfb754abfb88f198f886f50507ac not found: ID does not exist" Jul 12 19:42:05.723852 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:42:05.723833 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z"] Jul 12 19:42:05.727667 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:42:05.727648 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wbc8z"] Jul 12 19:42:05.916258 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:42:05.916226 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4" path="/var/lib/kubelet/pods/e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4/volumes" Jul 12 19:43:01.903703 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:01.903660 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:43:02.952440 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:02.952404 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftr4m"] Jul 12 19:43:02.952912 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:02.952806 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4" containerName="manager" Jul 12 19:43:02.952912 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:02.952819 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4" containerName="manager" Jul 12 19:43:02.953089 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:02.952930 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0634f8b-aaf5-4e41-b384-a8c9a2b2a4c4" containerName="manager" Jul 12 19:43:02.957177 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:02.957158 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftr4m" Jul 12 19:43:02.959548 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:02.959524 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz7xv\" (UniqueName: \"kubernetes.io/projected/1497e0f8-a76f-40a3-927f-c772158234d9-kube-api-access-rz7xv\") pod \"kuadrant-operator-controller-manager-86878c769f-ftr4m\" (UID: \"1497e0f8-a76f-40a3-927f-c772158234d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftr4m" Jul 12 19:43:02.959680 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:02.959662 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1497e0f8-a76f-40a3-927f-c772158234d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ftr4m\" (UID: \"1497e0f8-a76f-40a3-927f-c772158234d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftr4m" Jul 12 19:43:02.962069 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:02.962053 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kcwqf\"" Jul 12 19:43:02.967185 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:02.967166 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftr4m"] Jul 12 19:43:03.060681 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:03.060644 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1497e0f8-a76f-40a3-927f-c772158234d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ftr4m\" (UID: \"1497e0f8-a76f-40a3-927f-c772158234d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftr4m" Jul 12 19:43:03.060883 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:03.060696 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz7xv\" (UniqueName: \"kubernetes.io/projected/1497e0f8-a76f-40a3-927f-c772158234d9-kube-api-access-rz7xv\") pod \"kuadrant-operator-controller-manager-86878c769f-ftr4m\" (UID: \"1497e0f8-a76f-40a3-927f-c772158234d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftr4m" Jul 12 19:43:03.061068 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:03.061013 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1497e0f8-a76f-40a3-927f-c772158234d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ftr4m\" (UID: \"1497e0f8-a76f-40a3-927f-c772158234d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftr4m" Jul 12 19:43:03.069504 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:03.069484 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz7xv\" (UniqueName: \"kubernetes.io/projected/1497e0f8-a76f-40a3-927f-c772158234d9-kube-api-access-rz7xv\") pod \"kuadrant-operator-controller-manager-86878c769f-ftr4m\" (UID: \"1497e0f8-a76f-40a3-927f-c772158234d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftr4m" Jul 12 19:43:03.209153 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:03.209069 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:43:03.267386 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:03.267338 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftr4m" Jul 12 19:43:03.408950 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:03.408928 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftr4m"] Jul 12 19:43:03.411107 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:43:03.411081 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1497e0f8_a76f_40a3_927f_c772158234d9.slice/crio-22533201cf1721b4f91116321296f34be774ff1739cfce06b14567cc51f21e4d WatchSource:0}: Error finding container 22533201cf1721b4f91116321296f34be774ff1739cfce06b14567cc51f21e4d: Status 404 returned error can't find the container with id 22533201cf1721b4f91116321296f34be774ff1739cfce06b14567cc51f21e4d Jul 12 19:43:03.989617 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:03.989578 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftr4m" event={"ID":"1497e0f8-a76f-40a3-927f-c772158234d9","Type":"ContainerStarted","Data":"c90d711477e68859cee5bf15167a2ac82340ebb5ba1a6906958295735582f4ab"} Jul 12 19:43:03.990069 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:03.989640 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftr4m" event={"ID":"1497e0f8-a76f-40a3-927f-c772158234d9","Type":"ContainerStarted","Data":"22533201cf1721b4f91116321296f34be774ff1739cfce06b14567cc51f21e4d"} Jul 12 19:43:03.990069 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:03.989673 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftr4m" Jul 12 19:43:04.015716 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:04.015663 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftr4m" podStartSLOduration=2.015643606 podStartE2EDuration="2.015643606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:43:04.011539502 +0000 UTC m=+1392.604813602" watchObservedRunningTime="2026-07-12 19:43:04.015643606 +0000 UTC m=+1392.608917711" Jul 12 19:43:14.997543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:43:14.997499 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftr4m" Jul 12 19:44:29.296007 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:44:29.295972 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:44:29.395577 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:44:29.395544 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:03.588391 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:03.588351 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:12.832828 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:12.832789 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:12.933220 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:12.933184 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:13.046635 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:13.046603 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:21.297766 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:21.297724 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:26.369716 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:26.369679 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:29.074738 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:29.074702 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:29.172840 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:29.172808 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:29.507143 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:29.507108 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:35.125129 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.125096 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:35.183939 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.183906 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:35.297480 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.297449 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr"] Jul 12 19:49:35.300857 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.300837 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.303276 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.303255 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e8a6ab99-openshift-default-dockercfg-f9sts\"" Jul 12 19:49:35.319043 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.318920 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr"] Jul 12 19:49:35.323852 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.323791 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:35.383886 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.383796 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:35.408777 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.408741 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-token\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.408962 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.408797 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-credential-socket\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.408962 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.408824 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-podinfo\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.408962 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.408864 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e5917a-654e-4394-854f-28ca2bcc840d-istiod-ca-cert\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.408962 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.408895 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-data\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.408962 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.408918 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-workload-socket\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.408962 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.408957 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5wsz\" (UniqueName: \"kubernetes.io/projected/e9e5917a-654e-4394-854f-28ca2bcc840d-kube-api-access-x5wsz\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.409263 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.409045 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-workload-certs\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.409263 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.409076 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-envoy\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.510166 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.510116 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e5917a-654e-4394-854f-28ca2bcc840d-istiod-ca-cert\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.510166 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.510172 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-data\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.510439 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.510193 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-workload-socket\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.510439 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.510224 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5wsz\" (UniqueName: \"kubernetes.io/projected/e9e5917a-654e-4394-854f-28ca2bcc840d-kube-api-access-x5wsz\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.510439 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.510267 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-workload-certs\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.510439 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.510286 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-envoy\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.510439 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.510428 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-token\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.510681 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.510488 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-credential-socket\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.510681 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.510515 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-podinfo\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.510681 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.510665 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-workload-socket\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.510836 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.510807 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-data\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.510943 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.510876 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-workload-certs\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.510988 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.510963 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e5917a-654e-4394-854f-28ca2bcc840d-istiod-ca-cert\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.511070 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.511052 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-credential-socket\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.512978 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.512954 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-envoy\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.513531 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.513508 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-podinfo\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.518867 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.518833 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-token\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.519001 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.518984 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5wsz\" (UniqueName: \"kubernetes.io/projected/e9e5917a-654e-4394-854f-28ca2bcc840d-kube-api-access-x5wsz\") pod \"e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.612643 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.612610 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:35.758695 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.758663 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr"] Jul 12 19:49:35.760509 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:49:35.760482 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e5917a_654e_4394_854f_28ca2bcc840d.slice/crio-88aa08d49b28f85751a5b8e0ef332d2abbfde25aec8f2a41ea451e2394d5a228 WatchSource:0}: Error finding container 88aa08d49b28f85751a5b8e0ef332d2abbfde25aec8f2a41ea451e2394d5a228: Status 404 returned error can't find the container with id 88aa08d49b28f85751a5b8e0ef332d2abbfde25aec8f2a41ea451e2394d5a228 Jul 12 19:49:35.762509 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.762471 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:49:35.762600 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.762532 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:49:35.762600 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.762564 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:49:35.983331 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.983218 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" event={"ID":"e9e5917a-654e-4394-854f-28ca2bcc840d","Type":"ContainerStarted","Data":"76e1d12641ba6d9f29fe48374d8ce22380f7b28e9077a9e8887782f7867ee72a"} Jul 12 19:49:35.983331 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:35.983289 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" event={"ID":"e9e5917a-654e-4394-854f-28ca2bcc840d","Type":"ContainerStarted","Data":"88aa08d49b28f85751a5b8e0ef332d2abbfde25aec8f2a41ea451e2394d5a228"} Jul 12 19:49:36.002880 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:36.002831 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" podStartSLOduration=1.002813873 podStartE2EDuration="1.002813873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:49:36.000187688 +0000 UTC m=+1784.593461783" watchObservedRunningTime="2026-07-12 19:49:36.002813873 +0000 UTC m=+1784.596087971" Jul 12 19:49:36.513063 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:36.512974 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:36.593555 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:36.593524 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:36.613183 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:36.613152 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:36.619123 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:36.619086 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:36.656576 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:36.656538 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:36.757260 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:36.757214 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:36.989094 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:36.989057 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:36.989871 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:36.989852 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:37.154659 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:37.154626 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:37.257247 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:37.257151 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:38.820288 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:38.820245 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:41.516045 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:41.515990 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:41.676946 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:41.676910 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr"] Jul 12 19:49:41.677139 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:41.677119 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" podUID="e9e5917a-654e-4394-854f-28ca2bcc840d" containerName="istio-proxy" containerID="cri-o://76e1d12641ba6d9f29fe48374d8ce22380f7b28e9077a9e8887782f7867ee72a" gracePeriod=30 Jul 12 19:49:41.703112 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:41.703079 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:41.792627 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:41.792494 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:41.892348 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:41.892303 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:43.191826 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:43.191793 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:46.939349 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:46.939288 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:47.011113 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.011081 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-token\") pod \"e9e5917a-654e-4394-854f-28ca2bcc840d\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " Jul 12 19:49:47.011113 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.011120 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-data\") pod \"e9e5917a-654e-4394-854f-28ca2bcc840d\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " Jul 12 19:49:47.011339 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.011141 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e5917a-654e-4394-854f-28ca2bcc840d-istiod-ca-cert\") pod \"e9e5917a-654e-4394-854f-28ca2bcc840d\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " Jul 12 19:49:47.011339 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.011165 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-workload-certs\") pod \"e9e5917a-654e-4394-854f-28ca2bcc840d\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " Jul 12 19:49:47.011339 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.011192 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-podinfo\") pod \"e9e5917a-654e-4394-854f-28ca2bcc840d\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " Jul 12 19:49:47.011339 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.011215 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-workload-socket\") pod \"e9e5917a-654e-4394-854f-28ca2bcc840d\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " Jul 12 19:49:47.011339 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.011278 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5wsz\" (UniqueName: \"kubernetes.io/projected/e9e5917a-654e-4394-854f-28ca2bcc840d-kube-api-access-x5wsz\") pod \"e9e5917a-654e-4394-854f-28ca2bcc840d\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " Jul 12 19:49:47.011339 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.011327 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-credential-socket\") pod \"e9e5917a-654e-4394-854f-28ca2bcc840d\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " Jul 12 19:49:47.011630 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.011385 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-envoy\") pod \"e9e5917a-654e-4394-854f-28ca2bcc840d\" (UID: \"e9e5917a-654e-4394-854f-28ca2bcc840d\") " Jul 12 19:49:47.011630 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.011432 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-data" (OuterVolumeSpecName: "istio-data") pod "e9e5917a-654e-4394-854f-28ca2bcc840d" (UID: "e9e5917a-654e-4394-854f-28ca2bcc840d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:49:47.011630 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.011478 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e9e5917a-654e-4394-854f-28ca2bcc840d" (UID: "e9e5917a-654e-4394-854f-28ca2bcc840d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:49:47.011630 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.011548 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e5917a-654e-4394-854f-28ca2bcc840d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e9e5917a-654e-4394-854f-28ca2bcc840d" (UID: "e9e5917a-654e-4394-854f-28ca2bcc840d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:49:47.011849 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.011680 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e9e5917a-654e-4394-854f-28ca2bcc840d" (UID: "e9e5917a-654e-4394-854f-28ca2bcc840d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:49:47.011909 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.011847 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-data\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:49:47.011909 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.011869 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e9e5917a-654e-4394-854f-28ca2bcc840d" (UID: "e9e5917a-654e-4394-854f-28ca2bcc840d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:49:47.012229 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.012006 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e5917a-654e-4394-854f-28ca2bcc840d-istiod-ca-cert\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:49:47.012229 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.012072 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-workload-certs\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:49:47.013940 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.013910 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e9e5917a-654e-4394-854f-28ca2bcc840d" (UID: "e9e5917a-654e-4394-854f-28ca2bcc840d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 19:49:47.014119 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.014088 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e5917a-654e-4394-854f-28ca2bcc840d-kube-api-access-x5wsz" (OuterVolumeSpecName: "kube-api-access-x5wsz") pod "e9e5917a-654e-4394-854f-28ca2bcc840d" (UID: "e9e5917a-654e-4394-854f-28ca2bcc840d"). InnerVolumeSpecName "kube-api-access-x5wsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:49:47.014180 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.014098 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-token" (OuterVolumeSpecName: "istio-token") pod "e9e5917a-654e-4394-854f-28ca2bcc840d" (UID: "e9e5917a-654e-4394-854f-28ca2bcc840d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:49:47.014261 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.014241 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e9e5917a-654e-4394-854f-28ca2bcc840d" (UID: "e9e5917a-654e-4394-854f-28ca2bcc840d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:49:47.043161 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.043130 2564 generic.go:356] "Generic (PLEG): container finished" podID="e9e5917a-654e-4394-854f-28ca2bcc840d" containerID="76e1d12641ba6d9f29fe48374d8ce22380f7b28e9077a9e8887782f7867ee72a" exitCode=0 Jul 12 19:49:47.043294 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.043173 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" Jul 12 19:49:47.043294 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.043215 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" event={"ID":"e9e5917a-654e-4394-854f-28ca2bcc840d","Type":"ContainerDied","Data":"76e1d12641ba6d9f29fe48374d8ce22380f7b28e9077a9e8887782f7867ee72a"} Jul 12 19:49:47.043294 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.043265 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr" event={"ID":"e9e5917a-654e-4394-854f-28ca2bcc840d","Type":"ContainerDied","Data":"88aa08d49b28f85751a5b8e0ef332d2abbfde25aec8f2a41ea451e2394d5a228"} Jul 12 19:49:47.043294 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.043287 2564 scope.go:117] "RemoveContainer" containerID="76e1d12641ba6d9f29fe48374d8ce22380f7b28e9077a9e8887782f7867ee72a" Jul 12 19:49:47.052781 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.052765 2564 scope.go:117] "RemoveContainer" containerID="76e1d12641ba6d9f29fe48374d8ce22380f7b28e9077a9e8887782f7867ee72a" Jul 12 19:49:47.053063 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:49:47.053045 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e1d12641ba6d9f29fe48374d8ce22380f7b28e9077a9e8887782f7867ee72a\": container with ID starting with 76e1d12641ba6d9f29fe48374d8ce22380f7b28e9077a9e8887782f7867ee72a not found: ID does not exist" containerID="76e1d12641ba6d9f29fe48374d8ce22380f7b28e9077a9e8887782f7867ee72a" Jul 12 19:49:47.053122 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.053071 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e1d12641ba6d9f29fe48374d8ce22380f7b28e9077a9e8887782f7867ee72a"} err="failed to get container status \"76e1d12641ba6d9f29fe48374d8ce22380f7b28e9077a9e8887782f7867ee72a\": rpc error: code = NotFound desc = could not find container \"76e1d12641ba6d9f29fe48374d8ce22380f7b28e9077a9e8887782f7867ee72a\": container with ID starting with 76e1d12641ba6d9f29fe48374d8ce22380f7b28e9077a9e8887782f7867ee72a not found: ID does not exist" Jul 12 19:49:47.065247 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.065224 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr"] Jul 12 19:49:47.069853 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.069831 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e8a6ab99-openshift-default-8b758fc97-tn2rr"] Jul 12 19:49:47.113167 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.113138 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5wsz\" (UniqueName: \"kubernetes.io/projected/e9e5917a-654e-4394-854f-28ca2bcc840d-kube-api-access-x5wsz\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:49:47.113167 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.113167 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-credential-socket\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:49:47.113355 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.113178 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-envoy\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:49:47.113355 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.113187 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-token\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:49:47.113355 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.113195 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e5917a-654e-4394-854f-28ca2bcc840d-istio-podinfo\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:49:47.113355 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.113202 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e5917a-654e-4394-854f-28ca2bcc840d-workload-socket\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:49:47.500551 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.500512 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:47.558030 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.557989 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:47.660164 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.660118 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn"] Jul 12 19:49:47.660911 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.660874 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9e5917a-654e-4394-854f-28ca2bcc840d" containerName="istio-proxy" Jul 12 19:49:47.660911 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.660900 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9e5917a-654e-4394-854f-28ca2bcc840d" containerName="istio-proxy" Jul 12 19:49:47.661103 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.661008 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9e5917a-654e-4394-854f-28ca2bcc840d" containerName="istio-proxy" Jul 12 19:49:47.665883 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.665862 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.668416 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.668396 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1b63ffec-openshift-default-dockercfg-f6ncf\"" Jul 12 19:49:47.673521 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.673486 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn"] Jul 12 19:49:47.692583 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.692544 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:47.718601 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.718565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-token\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.718762 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.718607 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-envoy\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.718762 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.718626 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-podinfo\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.718762 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.718729 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-workload-certs\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.718894 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.718779 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-data\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.718894 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.718851 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istiod-ca-cert\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.718894 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.718869 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg9cl\" (UniqueName: \"kubernetes.io/projected/ca0c77ad-bc04-4f70-897c-48fdb01a7661-kube-api-access-bg9cl\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.719010 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.718902 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-workload-socket\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.719010 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.718926 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-credential-socket\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.792742 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.792649 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:47.820030 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.819985 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istiod-ca-cert\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.820177 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.820057 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg9cl\" (UniqueName: \"kubernetes.io/projected/ca0c77ad-bc04-4f70-897c-48fdb01a7661-kube-api-access-bg9cl\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.820177 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.820096 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-workload-socket\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.820177 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.820122 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-credential-socket\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.820177 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.820166 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-token\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.820380 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.820222 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-envoy\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.820380 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.820248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-podinfo\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.820560 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.820534 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-workload-socket\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.820775 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.820751 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-workload-certs\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.821075 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.820794 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-credential-socket\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.821075 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.821012 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-workload-certs\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.821075 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.821060 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-data\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.821268 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.821122 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istiod-ca-cert\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.821359 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.821342 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-data\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.823320 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.823299 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-envoy\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.823529 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.823512 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-podinfo\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.827929 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.827910 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-token\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.828071 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.828055 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg9cl\" (UniqueName: \"kubernetes.io/projected/ca0c77ad-bc04-4f70-897c-48fdb01a7661-kube-api-access-bg9cl\") pod \"e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:47.917430 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.917392 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9e5917a-654e-4394-854f-28ca2bcc840d" path="/var/lib/kubelet/pods/e9e5917a-654e-4394-854f-28ca2bcc840d/volumes" Jul 12 19:49:47.979177 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:47.979149 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:48.122356 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:48.122312 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn"] Jul 12 19:49:48.124142 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:49:48.124099 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0c77ad_bc04_4f70_897c_48fdb01a7661.slice/crio-b0f818c78b8a8159c183d10200ff770187a22aba1b64020340220ecc06a41e5d WatchSource:0}: Error finding container b0f818c78b8a8159c183d10200ff770187a22aba1b64020340220ecc06a41e5d: Status 404 returned error can't find the container with id b0f818c78b8a8159c183d10200ff770187a22aba1b64020340220ecc06a41e5d Jul 12 19:49:48.126478 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:48.126452 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:49:48.126552 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:48.126517 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:49:48.126552 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:48.126547 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:49:48.781748 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:48.781710 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:48.850098 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:48.850065 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:48.960722 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:48.960685 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:49.058539 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:49.058450 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" event={"ID":"ca0c77ad-bc04-4f70-897c-48fdb01a7661","Type":"ContainerStarted","Data":"40ebce1afede58fe89a6a718fb1667deb17b8bc00eb052f6e1a8673f45d1a83b"} Jul 12 19:49:49.058539 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:49.058512 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" event={"ID":"ca0c77ad-bc04-4f70-897c-48fdb01a7661","Type":"ContainerStarted","Data":"b0f818c78b8a8159c183d10200ff770187a22aba1b64020340220ecc06a41e5d"} Jul 12 19:49:49.084763 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:49.084707 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" podStartSLOduration=2.08469086 podStartE2EDuration="2.08469086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:49:49.083203833 +0000 UTC m=+1797.676477943" watchObservedRunningTime="2026-07-12 19:49:49.08469086 +0000 UTC m=+1797.677964957" Jul 12 19:49:49.133305 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:49.133270 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:49.419806 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:49.419776 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:49.520936 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:49.520896 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:49.979870 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:49.979835 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:49.984665 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:49.984643 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:50.069585 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:50.066100 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:50.070421 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:50.070400 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:49:50.991749 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:50.991709 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:53.844829 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:53.844790 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:54.253125 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:54.253084 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:59.926013 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:59.925978 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn"] Jul 12 19:49:59.926491 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:59.926185 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" podUID="ca0c77ad-bc04-4f70-897c-48fdb01a7661" containerName="istio-proxy" containerID="cri-o://40ebce1afede58fe89a6a718fb1667deb17b8bc00eb052f6e1a8673f45d1a83b" gracePeriod=30 Jul 12 19:49:59.947491 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:49:59.947457 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:00.050240 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.049922 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:00.071226 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.071200 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:50:00.119247 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.119211 2564 generic.go:356] "Generic (PLEG): container finished" podID="ca0c77ad-bc04-4f70-897c-48fdb01a7661" containerID="40ebce1afede58fe89a6a718fb1667deb17b8bc00eb052f6e1a8673f45d1a83b" exitCode=0 Jul 12 19:50:00.119444 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.119265 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" Jul 12 19:50:00.119444 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.119295 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" event={"ID":"ca0c77ad-bc04-4f70-897c-48fdb01a7661","Type":"ContainerDied","Data":"40ebce1afede58fe89a6a718fb1667deb17b8bc00eb052f6e1a8673f45d1a83b"} Jul 12 19:50:00.119444 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.119349 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn" event={"ID":"ca0c77ad-bc04-4f70-897c-48fdb01a7661","Type":"ContainerDied","Data":"b0f818c78b8a8159c183d10200ff770187a22aba1b64020340220ecc06a41e5d"} Jul 12 19:50:00.119444 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.119380 2564 scope.go:117] "RemoveContainer" containerID="40ebce1afede58fe89a6a718fb1667deb17b8bc00eb052f6e1a8673f45d1a83b" Jul 12 19:50:00.129573 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.129549 2564 scope.go:117] "RemoveContainer" containerID="40ebce1afede58fe89a6a718fb1667deb17b8bc00eb052f6e1a8673f45d1a83b" Jul 12 19:50:00.129903 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:50:00.129877 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ebce1afede58fe89a6a718fb1667deb17b8bc00eb052f6e1a8673f45d1a83b\": container with ID starting with 40ebce1afede58fe89a6a718fb1667deb17b8bc00eb052f6e1a8673f45d1a83b not found: ID does not exist" containerID="40ebce1afede58fe89a6a718fb1667deb17b8bc00eb052f6e1a8673f45d1a83b" Jul 12 19:50:00.130001 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.129918 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ebce1afede58fe89a6a718fb1667deb17b8bc00eb052f6e1a8673f45d1a83b"} err="failed to get container status \"40ebce1afede58fe89a6a718fb1667deb17b8bc00eb052f6e1a8673f45d1a83b\": rpc error: code = NotFound desc = could not find container \"40ebce1afede58fe89a6a718fb1667deb17b8bc00eb052f6e1a8673f45d1a83b\": container with ID starting with 40ebce1afede58fe89a6a718fb1667deb17b8bc00eb052f6e1a8673f45d1a83b not found: ID does not exist" Jul 12 19:50:00.138207 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.138185 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bg9cl\" (UniqueName: \"kubernetes.io/projected/ca0c77ad-bc04-4f70-897c-48fdb01a7661-kube-api-access-bg9cl\") pod \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " Jul 12 19:50:00.138333 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.138224 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-workload-certs\") pod \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " Jul 12 19:50:00.138333 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.138246 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-credential-socket\") pod \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " Jul 12 19:50:00.138333 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.138307 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-token\") pod \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " Jul 12 19:50:00.138333 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.138329 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-podinfo\") pod \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " Jul 12 19:50:00.138553 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.138363 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istiod-ca-cert\") pod \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " Jul 12 19:50:00.138553 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.138395 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-workload-socket\") pod \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " Jul 12 19:50:00.138553 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.138434 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-data\") pod \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " Jul 12 19:50:00.138553 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.138461 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-envoy\") pod \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\" (UID: \"ca0c77ad-bc04-4f70-897c-48fdb01a7661\") " Jul 12 19:50:00.138756 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.138560 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ca0c77ad-bc04-4f70-897c-48fdb01a7661" (UID: "ca0c77ad-bc04-4f70-897c-48fdb01a7661"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:50:00.138756 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.138626 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ca0c77ad-bc04-4f70-897c-48fdb01a7661" (UID: "ca0c77ad-bc04-4f70-897c-48fdb01a7661"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:50:00.138871 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.138756 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ca0c77ad-bc04-4f70-897c-48fdb01a7661" (UID: "ca0c77ad-bc04-4f70-897c-48fdb01a7661"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:50:00.138871 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.138788 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-workload-certs\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:50:00.138871 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.138807 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-credential-socket\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:50:00.139072 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.138876 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ca0c77ad-bc04-4f70-897c-48fdb01a7661" (UID: "ca0c77ad-bc04-4f70-897c-48fdb01a7661"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:50:00.139629 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.139601 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-data" (OuterVolumeSpecName: "istio-data") pod "ca0c77ad-bc04-4f70-897c-48fdb01a7661" (UID: "ca0c77ad-bc04-4f70-897c-48fdb01a7661"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:50:00.140667 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.140645 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0c77ad-bc04-4f70-897c-48fdb01a7661-kube-api-access-bg9cl" (OuterVolumeSpecName: "kube-api-access-bg9cl") pod "ca0c77ad-bc04-4f70-897c-48fdb01a7661" (UID: "ca0c77ad-bc04-4f70-897c-48fdb01a7661"). InnerVolumeSpecName "kube-api-access-bg9cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:50:00.140949 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.140927 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-token" (OuterVolumeSpecName: "istio-token") pod "ca0c77ad-bc04-4f70-897c-48fdb01a7661" (UID: "ca0c77ad-bc04-4f70-897c-48fdb01a7661"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:50:00.141205 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.141179 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ca0c77ad-bc04-4f70-897c-48fdb01a7661" (UID: "ca0c77ad-bc04-4f70-897c-48fdb01a7661"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:50:00.141301 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.141265 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ca0c77ad-bc04-4f70-897c-48fdb01a7661" (UID: "ca0c77ad-bc04-4f70-897c-48fdb01a7661"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 19:50:00.239461 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.239426 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-token\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:50:00.239461 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.239459 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-podinfo\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:50:00.239461 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.239469 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istiod-ca-cert\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:50:00.239693 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.239478 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-workload-socket\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:50:00.239693 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.239488 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-data\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:50:00.239693 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.239496 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0c77ad-bc04-4f70-897c-48fdb01a7661-istio-envoy\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:50:00.239693 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.239504 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg9cl\" (UniqueName: \"kubernetes.io/projected/ca0c77ad-bc04-4f70-897c-48fdb01a7661-kube-api-access-bg9cl\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:50:00.442551 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.442524 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn"] Jul 12 19:50:00.446621 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:00.446597 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1b63ffec-openshift-default-7f4965754c-gd7zn"] Jul 12 19:50:01.917065 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:01.917006 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca0c77ad-bc04-4f70-897c-48fdb01a7661" path="/var/lib/kubelet/pods/ca0c77ad-bc04-4f70-897c-48fdb01a7661/volumes" Jul 12 19:50:02.242231 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:02.242199 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:05.793699 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:05.793662 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:05.893741 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:05.893702 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:05.965757 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:05.965723 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6"] Jul 12 19:50:05.966133 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:05.966118 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca0c77ad-bc04-4f70-897c-48fdb01a7661" containerName="istio-proxy" Jul 12 19:50:05.966208 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:05.966137 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca0c77ad-bc04-4f70-897c-48fdb01a7661" containerName="istio-proxy" Jul 12 19:50:05.966208 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:05.966203 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca0c77ad-bc04-4f70-897c-48fdb01a7661" containerName="istio-proxy" Jul 12 19:50:05.970706 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:05.970685 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:05.973137 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:05.973111 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-9fa1f168-openshift-default-dockercfg-8pn9v\"" Jul 12 19:50:05.981080 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:05.981012 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6"] Jul 12 19:50:06.000191 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.000161 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:06.093093 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.092996 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-credential-socket\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.093093 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.093061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-workload-certs\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.093272 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.093109 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-istio-envoy\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.093272 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.093155 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kxwt\" (UniqueName: \"kubernetes.io/projected/8822536b-298a-4ade-a694-8a54a78db19a-kube-api-access-9kxwt\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.093340 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.093286 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-workload-socket\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.093340 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.093324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-istio-data\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.093407 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.093366 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8822536b-298a-4ade-a694-8a54a78db19a-istio-token\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.093407 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.093385 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8822536b-298a-4ade-a694-8a54a78db19a-istio-podinfo\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.093407 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.093400 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8822536b-298a-4ade-a694-8a54a78db19a-istiod-ca-cert\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.097383 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.097342 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:06.194276 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.194243 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-credential-socket\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.194276 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.194278 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-workload-certs\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.194498 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.194297 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-istio-envoy\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.194498 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.194411 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kxwt\" (UniqueName: \"kubernetes.io/projected/8822536b-298a-4ade-a694-8a54a78db19a-kube-api-access-9kxwt\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.194575 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.194509 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-workload-socket\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.194575 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.194535 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-istio-data\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.194681 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.194572 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8822536b-298a-4ade-a694-8a54a78db19a-istio-token\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.194681 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.194599 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8822536b-298a-4ade-a694-8a54a78db19a-istio-podinfo\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.194681 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.194628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8822536b-298a-4ade-a694-8a54a78db19a-istiod-ca-cert\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.194681 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.194650 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-credential-socket\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.194890 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.194712 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-workload-certs\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.194953 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.194887 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-workload-socket\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.194953 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.194912 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-istio-data\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.195595 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.195569 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8822536b-298a-4ade-a694-8a54a78db19a-istiod-ca-cert\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.196935 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.196910 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-istio-envoy\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.197174 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.197158 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8822536b-298a-4ade-a694-8a54a78db19a-istio-podinfo\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.202135 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.202114 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8822536b-298a-4ade-a694-8a54a78db19a-istio-token\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.202519 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.202499 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kxwt\" (UniqueName: \"kubernetes.io/projected/8822536b-298a-4ade-a694-8a54a78db19a-kube-api-access-9kxwt\") pod \"e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.282902 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.282872 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:06.431404 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.431377 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6"] Jul 12 19:50:06.433723 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:50:06.433697 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8822536b_298a_4ade_a694_8a54a78db19a.slice/crio-2244cfb760addbf2e665ffc31dfbe9f64428a552a8f5ec8d4284143804f242b8 WatchSource:0}: Error finding container 2244cfb760addbf2e665ffc31dfbe9f64428a552a8f5ec8d4284143804f242b8: Status 404 returned error can't find the container with id 2244cfb760addbf2e665ffc31dfbe9f64428a552a8f5ec8d4284143804f242b8 Jul 12 19:50:06.435926 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.435893 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:50:06.436007 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.435968 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:50:06.436007 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:06.435999 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:50:07.092285 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:07.092249 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:07.157717 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:07.157681 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" event={"ID":"8822536b-298a-4ade-a694-8a54a78db19a","Type":"ContainerStarted","Data":"497d0b79418ddf023c7b1b5e082bd3c780ed00a2376c188316f31614512138d2"} Jul 12 19:50:07.157888 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:07.157783 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" event={"ID":"8822536b-298a-4ade-a694-8a54a78db19a","Type":"ContainerStarted","Data":"2244cfb760addbf2e665ffc31dfbe9f64428a552a8f5ec8d4284143804f242b8"} Jul 12 19:50:07.171806 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:07.171774 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:07.185182 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:07.185136 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" podStartSLOduration=2.185115855 podStartE2EDuration="2.185115855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:50:07.181815871 +0000 UTC m=+1815.775089963" watchObservedRunningTime="2026-07-12 19:50:07.185115855 +0000 UTC m=+1815.778389953" Jul 12 19:50:07.237439 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:07.237408 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:07.283544 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:07.283510 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:07.288406 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:07.288384 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:07.477700 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:07.477659 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:07.740233 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:07.740146 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:07.836782 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:07.836749 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:08.164228 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:08.164203 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:08.165164 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:08.165136 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:09.251093 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:09.251051 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:11.518074 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:11.518034 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:11.684239 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:11.684194 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6"] Jul 12 19:50:11.684458 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:11.684432 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" podUID="8822536b-298a-4ade-a694-8a54a78db19a" containerName="istio-proxy" containerID="cri-o://497d0b79418ddf023c7b1b5e082bd3c780ed00a2376c188316f31614512138d2" gracePeriod=30 Jul 12 19:50:11.711749 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:11.711712 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:11.801194 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:11.801104 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:11.875220 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:11.875184 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:13.576683 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:13.576652 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:16.939363 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:16.939339 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:17.094730 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.094648 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-workload-socket\") pod \"8822536b-298a-4ade-a694-8a54a78db19a\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " Jul 12 19:50:17.094730 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.094679 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-istio-data\") pod \"8822536b-298a-4ade-a694-8a54a78db19a\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " Jul 12 19:50:17.094730 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.094714 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8822536b-298a-4ade-a694-8a54a78db19a-istiod-ca-cert\") pod \"8822536b-298a-4ade-a694-8a54a78db19a\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " Jul 12 19:50:17.095062 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.094738 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kxwt\" (UniqueName: \"kubernetes.io/projected/8822536b-298a-4ade-a694-8a54a78db19a-kube-api-access-9kxwt\") pod \"8822536b-298a-4ade-a694-8a54a78db19a\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " Jul 12 19:50:17.095062 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.094786 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8822536b-298a-4ade-a694-8a54a78db19a-istio-token\") pod \"8822536b-298a-4ade-a694-8a54a78db19a\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " Jul 12 19:50:17.095062 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.094806 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-workload-certs\") pod \"8822536b-298a-4ade-a694-8a54a78db19a\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " Jul 12 19:50:17.095062 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.094848 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-credential-socket\") pod \"8822536b-298a-4ade-a694-8a54a78db19a\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " Jul 12 19:50:17.095062 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.094880 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-istio-envoy\") pod \"8822536b-298a-4ade-a694-8a54a78db19a\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " Jul 12 19:50:17.095062 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.094909 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8822536b-298a-4ade-a694-8a54a78db19a" (UID: "8822536b-298a-4ade-a694-8a54a78db19a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:50:17.095062 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.094927 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8822536b-298a-4ade-a694-8a54a78db19a-istio-podinfo\") pod \"8822536b-298a-4ade-a694-8a54a78db19a\" (UID: \"8822536b-298a-4ade-a694-8a54a78db19a\") " Jul 12 19:50:17.095432 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.095070 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-istio-data" (OuterVolumeSpecName: "istio-data") pod "8822536b-298a-4ade-a694-8a54a78db19a" (UID: "8822536b-298a-4ade-a694-8a54a78db19a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:50:17.095432 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.095157 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8822536b-298a-4ade-a694-8a54a78db19a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8822536b-298a-4ade-a694-8a54a78db19a" (UID: "8822536b-298a-4ade-a694-8a54a78db19a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:50:17.095432 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.095185 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8822536b-298a-4ade-a694-8a54a78db19a" (UID: "8822536b-298a-4ade-a694-8a54a78db19a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:50:17.095432 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.095282 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8822536b-298a-4ade-a694-8a54a78db19a" (UID: "8822536b-298a-4ade-a694-8a54a78db19a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:50:17.095432 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.095367 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-credential-socket\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:50:17.095432 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.095384 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-workload-socket\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:50:17.095432 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.095399 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-istio-data\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:50:17.095432 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.095413 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8822536b-298a-4ade-a694-8a54a78db19a-istiod-ca-cert\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:50:17.095432 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.095428 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-workload-certs\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:50:17.097244 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.097221 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8822536b-298a-4ade-a694-8a54a78db19a-istio-token" (OuterVolumeSpecName: "istio-token") pod "8822536b-298a-4ade-a694-8a54a78db19a" (UID: "8822536b-298a-4ade-a694-8a54a78db19a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:50:17.097346 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.097272 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8822536b-298a-4ade-a694-8a54a78db19a-kube-api-access-9kxwt" (OuterVolumeSpecName: "kube-api-access-9kxwt") pod "8822536b-298a-4ade-a694-8a54a78db19a" (UID: "8822536b-298a-4ade-a694-8a54a78db19a"). InnerVolumeSpecName "kube-api-access-9kxwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:50:17.097581 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.097561 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8822536b-298a-4ade-a694-8a54a78db19a" (UID: "8822536b-298a-4ade-a694-8a54a78db19a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:50:17.097660 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.097637 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8822536b-298a-4ade-a694-8a54a78db19a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8822536b-298a-4ade-a694-8a54a78db19a" (UID: "8822536b-298a-4ade-a694-8a54a78db19a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 19:50:17.196037 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.195979 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kxwt\" (UniqueName: \"kubernetes.io/projected/8822536b-298a-4ade-a694-8a54a78db19a-kube-api-access-9kxwt\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:50:17.196037 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.196010 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8822536b-298a-4ade-a694-8a54a78db19a-istio-token\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:50:17.196037 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.196037 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8822536b-298a-4ade-a694-8a54a78db19a-istio-envoy\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:50:17.196037 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.196047 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8822536b-298a-4ade-a694-8a54a78db19a-istio-podinfo\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:50:17.209638 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.209606 2564 generic.go:356] "Generic (PLEG): container finished" podID="8822536b-298a-4ade-a694-8a54a78db19a" containerID="497d0b79418ddf023c7b1b5e082bd3c780ed00a2376c188316f31614512138d2" exitCode=0 Jul 12 19:50:17.209801 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.209653 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" Jul 12 19:50:17.209801 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.209694 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" event={"ID":"8822536b-298a-4ade-a694-8a54a78db19a","Type":"ContainerDied","Data":"497d0b79418ddf023c7b1b5e082bd3c780ed00a2376c188316f31614512138d2"} Jul 12 19:50:17.209801 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.209758 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6" event={"ID":"8822536b-298a-4ade-a694-8a54a78db19a","Type":"ContainerDied","Data":"2244cfb760addbf2e665ffc31dfbe9f64428a552a8f5ec8d4284143804f242b8"} Jul 12 19:50:17.209801 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.209788 2564 scope.go:117] "RemoveContainer" containerID="497d0b79418ddf023c7b1b5e082bd3c780ed00a2376c188316f31614512138d2" Jul 12 19:50:17.219454 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.219339 2564 scope.go:117] "RemoveContainer" containerID="497d0b79418ddf023c7b1b5e082bd3c780ed00a2376c188316f31614512138d2" Jul 12 19:50:17.219667 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:50:17.219638 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"497d0b79418ddf023c7b1b5e082bd3c780ed00a2376c188316f31614512138d2\": container with ID starting with 497d0b79418ddf023c7b1b5e082bd3c780ed00a2376c188316f31614512138d2 not found: ID does not exist" containerID="497d0b79418ddf023c7b1b5e082bd3c780ed00a2376c188316f31614512138d2" Jul 12 19:50:17.219760 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.219672 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"497d0b79418ddf023c7b1b5e082bd3c780ed00a2376c188316f31614512138d2"} err="failed to get container status \"497d0b79418ddf023c7b1b5e082bd3c780ed00a2376c188316f31614512138d2\": rpc error: code = NotFound desc = could not find container \"497d0b79418ddf023c7b1b5e082bd3c780ed00a2376c188316f31614512138d2\": container with ID starting with 497d0b79418ddf023c7b1b5e082bd3c780ed00a2376c188316f31614512138d2 not found: ID does not exist" Jul 12 19:50:17.240206 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.237689 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6"] Jul 12 19:50:17.243171 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.243149 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-9fa1f168-openshift-default-85c847755f-jt5h6"] Jul 12 19:50:17.591174 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.591136 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:17.691697 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.691663 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:17.917246 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:17.917216 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8822536b-298a-4ade-a694-8a54a78db19a" path="/var/lib/kubelet/pods/8822536b-298a-4ade-a694-8a54a78db19a/volumes" Jul 12 19:50:18.026776 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:18.026735 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:18.127481 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:18.127443 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:18.226185 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:18.226075 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:19.559233 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:19.559198 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:25.160813 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:25.160769 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:25.261651 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:25.261617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:25.368571 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:25.368537 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:25.488569 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:25.488470 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:25.862178 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:25.862078 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:51.807586 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:51.807550 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:57.395910 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:57.395873 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:57.467436 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:57.467400 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:57.543184 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:57.543149 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:57.652859 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:50:57.652800 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:13.465527 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:13.465435 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:19.179597 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:19.179551 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:19.241194 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:19.241147 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:19.320404 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:19.320372 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:19.420169 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:19.420137 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:29.868884 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:29.868844 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:30.668251 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:30.668202 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:36.311220 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:36.311186 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:36.372932 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:36.372892 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:36.451726 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:36.451694 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:36.550001 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:36.549968 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:36.650428 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:36.650322 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:37.310494 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:37.310459 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:37.407797 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:37.407764 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:37.507596 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:37.507555 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:39.047360 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:39.047323 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:44.708481 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:44.708447 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:44.770241 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:44.770211 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:45.356430 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:45.356393 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:51.081832 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.081793 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:51.188212 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.188171 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:51.288907 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.288868 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:51.521826 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.521791 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b"] Jul 12 19:51:51.522300 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.522279 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8822536b-298a-4ade-a694-8a54a78db19a" containerName="istio-proxy" Jul 12 19:51:51.522300 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.522302 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8822536b-298a-4ade-a694-8a54a78db19a" containerName="istio-proxy" Jul 12 19:51:51.522469 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.522402 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8822536b-298a-4ade-a694-8a54a78db19a" containerName="istio-proxy" Jul 12 19:51:51.525466 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.525439 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.527818 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.527795 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ac7c507d-openshift-default-dockercfg-wwncp\"" Jul 12 19:51:51.549468 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.549442 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b"] Jul 12 19:51:51.549921 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.549892 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:51.604783 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.604746 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-workload-certs\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.604783 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.604785 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtpr4\" (UniqueName: \"kubernetes.io/projected/ba071a46-6981-4c45-9b12-d9ff4d465a85-kube-api-access-wtpr4\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.604981 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.604807 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-envoy\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.604981 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.604877 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-token\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.604981 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.604931 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-podinfo\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.604981 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.604949 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba071a46-6981-4c45-9b12-d9ff4d465a85-istiod-ca-cert\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.605191 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.605013 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-credential-socket\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.605191 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.605082 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-data\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.605191 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.605134 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-workload-socket\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.706334 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.706300 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-workload-socket\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.706521 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.706342 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-workload-certs\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.706521 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.706364 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtpr4\" (UniqueName: \"kubernetes.io/projected/ba071a46-6981-4c45-9b12-d9ff4d465a85-kube-api-access-wtpr4\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.706521 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.706394 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-envoy\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.706521 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.706424 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-token\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.706521 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.706466 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-podinfo\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.706521 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.706487 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba071a46-6981-4c45-9b12-d9ff4d465a85-istiod-ca-cert\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.706842 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.706539 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-credential-socket\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.706842 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.706679 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-data\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.706842 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.706753 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-workload-socket\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.707002 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.706853 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-credential-socket\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.707002 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.706960 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-data\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.707106 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.707083 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-workload-certs\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.707593 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.707561 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba071a46-6981-4c45-9b12-d9ff4d465a85-istiod-ca-cert\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.709054 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.709008 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-envoy\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.709264 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.709248 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-podinfo\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.714380 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.714357 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-token\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.714652 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.714633 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtpr4\" (UniqueName: \"kubernetes.io/projected/ba071a46-6981-4c45-9b12-d9ff4d465a85-kube-api-access-wtpr4\") pod \"e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.838667 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.838590 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ac7c507d-openshift-default-dockercfg-wwncp\"" Jul 12 19:51:51.846567 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.846538 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:51.993857 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.993835 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b"] Jul 12 19:51:51.996101 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:51:51.996071 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba071a46_6981_4c45_9b12_d9ff4d465a85.slice/crio-37b38ebb3cf0b8582a596c7cfa03c067de1b76850743b77dce8683a10c2be88f WatchSource:0}: Error finding container 37b38ebb3cf0b8582a596c7cfa03c067de1b76850743b77dce8683a10c2be88f: Status 404 returned error can't find the container with id 37b38ebb3cf0b8582a596c7cfa03c067de1b76850743b77dce8683a10c2be88f Jul 12 19:51:51.998365 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.998328 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:51:51.998449 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.998412 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:51:51.998493 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:51.998454 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:51:52.080497 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:52.080467 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:52.694997 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:52.694959 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" event={"ID":"ba071a46-6981-4c45-9b12-d9ff4d465a85","Type":"ContainerStarted","Data":"482a35874e90e9f74c83555e496e743858b24a61b8b84a1f4232cb8d473a5f4d"} Jul 12 19:51:52.695425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:52.695041 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" event={"ID":"ba071a46-6981-4c45-9b12-d9ff4d465a85","Type":"ContainerStarted","Data":"37b38ebb3cf0b8582a596c7cfa03c067de1b76850743b77dce8683a10c2be88f"} Jul 12 19:51:52.715346 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:52.715288 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" podStartSLOduration=1.715267578 podStartE2EDuration="1.715267578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:51:52.713786153 +0000 UTC m=+1921.307060264" watchObservedRunningTime="2026-07-12 19:51:52.715267578 +0000 UTC m=+1921.308541677" Jul 12 19:51:52.847068 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:52.847036 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:53.852656 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:53.852629 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:54.580955 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:54.580907 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:54.653190 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:54.653155 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:54.704379 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:54.704339 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:54.705374 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:54.705355 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:51:54.717075 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:54.717043 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:54.889240 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:54.889115 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:55.132537 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:55.132501 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:55.233370 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:55.233337 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:59.081452 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:59.081420 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:59.721545 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:59.721513 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:59.819150 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:51:59.819111 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:00.134221 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:00.134096 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:05.807675 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:05.807635 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:05.908632 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:05.908596 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:06.353276 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:06.353240 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:06.641801 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:06.641716 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:06.742450 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:06.742411 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:07.775371 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:07.775292 2564 generic.go:356] "Generic (PLEG): container finished" podID="ba071a46-6981-4c45-9b12-d9ff4d465a85" containerID="482a35874e90e9f74c83555e496e743858b24a61b8b84a1f4232cb8d473a5f4d" exitCode=0 Jul 12 19:52:07.775766 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:07.775369 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" event={"ID":"ba071a46-6981-4c45-9b12-d9ff4d465a85","Type":"ContainerDied","Data":"482a35874e90e9f74c83555e496e743858b24a61b8b84a1f4232cb8d473a5f4d"} Jul 12 19:52:07.775810 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:07.775775 2564 scope.go:117] "RemoveContainer" containerID="482a35874e90e9f74c83555e496e743858b24a61b8b84a1f4232cb8d473a5f4d" Jul 12 19:52:07.776769 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:07.776744 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:52:07.776825 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:07.776813 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:52:07.776864 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:07.776851 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:52:08.782639 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:08.782604 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" event={"ID":"ba071a46-6981-4c45-9b12-d9ff4d465a85","Type":"ContainerStarted","Data":"dd32d9025f0b12199335a1b1178b302cf5f0a8685406f0a62c926f0e53a4c532"} Jul 12 19:52:08.847640 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:08.847600 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:52:08.848752 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:08.848723 2564 patch_prober.go:28] interesting pod/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 12 19:52:08.848870 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:08.848779 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" podUID="ba071a46-6981-4c45-9b12-d9ff4d465a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 12 19:52:09.704353 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:09.704320 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:52:09.847822 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:09.847778 2564 patch_prober.go:28] interesting pod/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 12 19:52:09.848285 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:09.847845 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" podUID="ba071a46-6981-4c45-9b12-d9ff4d465a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 12 19:52:10.847463 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:10.847427 2564 patch_prober.go:28] interesting pod/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 12 19:52:10.847662 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:10.847497 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" podUID="ba071a46-6981-4c45-9b12-d9ff4d465a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 12 19:52:11.847195 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:11.847154 2564 patch_prober.go:28] interesting pod/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 12 19:52:11.847605 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:11.847216 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" podUID="ba071a46-6981-4c45-9b12-d9ff4d465a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 12 19:52:12.306506 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:12.306467 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:12.356486 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:12.356441 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:12.527299 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:12.527261 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b"] Jul 12 19:52:12.527535 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:12.527502 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" podUID="ba071a46-6981-4c45-9b12-d9ff4d465a85" containerName="istio-proxy" containerID="cri-o://dd32d9025f0b12199335a1b1178b302cf5f0a8685406f0a62c926f0e53a4c532" gracePeriod=30 Jul 12 19:52:12.542996 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:12.542956 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:12.885953 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:12.885916 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:12.982067 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:12.982014 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:13.085279 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:13.085248 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:17.787336 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.787306 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:52:17.832940 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.832855 2564 generic.go:356] "Generic (PLEG): container finished" podID="ba071a46-6981-4c45-9b12-d9ff4d465a85" containerID="dd32d9025f0b12199335a1b1178b302cf5f0a8685406f0a62c926f0e53a4c532" exitCode=0 Jul 12 19:52:17.832940 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.832912 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" Jul 12 19:52:17.833145 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.832949 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" event={"ID":"ba071a46-6981-4c45-9b12-d9ff4d465a85","Type":"ContainerDied","Data":"dd32d9025f0b12199335a1b1178b302cf5f0a8685406f0a62c926f0e53a4c532"} Jul 12 19:52:17.833145 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.833002 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b" event={"ID":"ba071a46-6981-4c45-9b12-d9ff4d465a85","Type":"ContainerDied","Data":"37b38ebb3cf0b8582a596c7cfa03c067de1b76850743b77dce8683a10c2be88f"} Jul 12 19:52:17.833145 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.833051 2564 scope.go:117] "RemoveContainer" containerID="dd32d9025f0b12199335a1b1178b302cf5f0a8685406f0a62c926f0e53a4c532" Jul 12 19:52:17.841741 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.841724 2564 scope.go:117] "RemoveContainer" containerID="482a35874e90e9f74c83555e496e743858b24a61b8b84a1f4232cb8d473a5f4d" Jul 12 19:52:17.849537 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.849519 2564 scope.go:117] "RemoveContainer" containerID="dd32d9025f0b12199335a1b1178b302cf5f0a8685406f0a62c926f0e53a4c532" Jul 12 19:52:17.849792 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:52:17.849775 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd32d9025f0b12199335a1b1178b302cf5f0a8685406f0a62c926f0e53a4c532\": container with ID starting with dd32d9025f0b12199335a1b1178b302cf5f0a8685406f0a62c926f0e53a4c532 not found: ID does not exist" containerID="dd32d9025f0b12199335a1b1178b302cf5f0a8685406f0a62c926f0e53a4c532" Jul 12 19:52:17.849834 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.849802 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd32d9025f0b12199335a1b1178b302cf5f0a8685406f0a62c926f0e53a4c532"} err="failed to get container status \"dd32d9025f0b12199335a1b1178b302cf5f0a8685406f0a62c926f0e53a4c532\": rpc error: code = NotFound desc = could not find container \"dd32d9025f0b12199335a1b1178b302cf5f0a8685406f0a62c926f0e53a4c532\": container with ID starting with dd32d9025f0b12199335a1b1178b302cf5f0a8685406f0a62c926f0e53a4c532 not found: ID does not exist" Jul 12 19:52:17.849834 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.849821 2564 scope.go:117] "RemoveContainer" containerID="482a35874e90e9f74c83555e496e743858b24a61b8b84a1f4232cb8d473a5f4d" Jul 12 19:52:17.850071 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:52:17.850054 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"482a35874e90e9f74c83555e496e743858b24a61b8b84a1f4232cb8d473a5f4d\": container with ID starting with 482a35874e90e9f74c83555e496e743858b24a61b8b84a1f4232cb8d473a5f4d not found: ID does not exist" containerID="482a35874e90e9f74c83555e496e743858b24a61b8b84a1f4232cb8d473a5f4d" Jul 12 19:52:17.850119 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.850077 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"482a35874e90e9f74c83555e496e743858b24a61b8b84a1f4232cb8d473a5f4d"} err="failed to get container status \"482a35874e90e9f74c83555e496e743858b24a61b8b84a1f4232cb8d473a5f4d\": rpc error: code = NotFound desc = could not find container \"482a35874e90e9f74c83555e496e743858b24a61b8b84a1f4232cb8d473a5f4d\": container with ID starting with 482a35874e90e9f74c83555e496e743858b24a61b8b84a1f4232cb8d473a5f4d not found: ID does not exist" Jul 12 19:52:17.960146 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.960115 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-data\") pod \"ba071a46-6981-4c45-9b12-d9ff4d465a85\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " Jul 12 19:52:17.960299 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.960158 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba071a46-6981-4c45-9b12-d9ff4d465a85-istiod-ca-cert\") pod \"ba071a46-6981-4c45-9b12-d9ff4d465a85\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " Jul 12 19:52:17.960299 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.960205 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-credential-socket\") pod \"ba071a46-6981-4c45-9b12-d9ff4d465a85\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " Jul 12 19:52:17.960299 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.960221 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-workload-socket\") pod \"ba071a46-6981-4c45-9b12-d9ff4d465a85\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " Jul 12 19:52:17.960299 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.960240 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-envoy\") pod \"ba071a46-6981-4c45-9b12-d9ff4d465a85\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " Jul 12 19:52:17.960299 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.960259 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtpr4\" (UniqueName: \"kubernetes.io/projected/ba071a46-6981-4c45-9b12-d9ff4d465a85-kube-api-access-wtpr4\") pod \"ba071a46-6981-4c45-9b12-d9ff4d465a85\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " Jul 12 19:52:17.960299 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.960275 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-token\") pod \"ba071a46-6981-4c45-9b12-d9ff4d465a85\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " Jul 12 19:52:17.960603 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.960524 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ba071a46-6981-4c45-9b12-d9ff4d465a85" (UID: "ba071a46-6981-4c45-9b12-d9ff4d465a85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:52:17.960678 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.960601 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-podinfo\") pod \"ba071a46-6981-4c45-9b12-d9ff4d465a85\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " Jul 12 19:52:17.960735 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.960681 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-workload-certs\") pod \"ba071a46-6981-4c45-9b12-d9ff4d465a85\" (UID: \"ba071a46-6981-4c45-9b12-d9ff4d465a85\") " Jul 12 19:52:17.960735 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.960704 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba071a46-6981-4c45-9b12-d9ff4d465a85-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ba071a46-6981-4c45-9b12-d9ff4d465a85" (UID: "ba071a46-6981-4c45-9b12-d9ff4d465a85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:52:17.960865 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.960806 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ba071a46-6981-4c45-9b12-d9ff4d465a85" (UID: "ba071a46-6981-4c45-9b12-d9ff4d465a85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:52:17.960934 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.960908 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-data" (OuterVolumeSpecName: "istio-data") pod "ba071a46-6981-4c45-9b12-d9ff4d465a85" (UID: "ba071a46-6981-4c45-9b12-d9ff4d465a85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:52:17.961066 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.961013 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ba071a46-6981-4c45-9b12-d9ff4d465a85" (UID: "ba071a46-6981-4c45-9b12-d9ff4d465a85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:52:17.961195 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.961114 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-data\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:52:17.961195 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.961134 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba071a46-6981-4c45-9b12-d9ff4d465a85-istiod-ca-cert\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:52:17.961195 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.961147 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-credential-socket\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:52:17.961195 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.961161 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-workload-socket\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:52:17.962809 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.962783 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ba071a46-6981-4c45-9b12-d9ff4d465a85" (UID: "ba071a46-6981-4c45-9b12-d9ff4d465a85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 19:52:17.962895 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.962847 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba071a46-6981-4c45-9b12-d9ff4d465a85-kube-api-access-wtpr4" (OuterVolumeSpecName: "kube-api-access-wtpr4") pod "ba071a46-6981-4c45-9b12-d9ff4d465a85" (UID: "ba071a46-6981-4c45-9b12-d9ff4d465a85"). InnerVolumeSpecName "kube-api-access-wtpr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:52:17.963079 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.963060 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ba071a46-6981-4c45-9b12-d9ff4d465a85" (UID: "ba071a46-6981-4c45-9b12-d9ff4d465a85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:52:17.963079 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:17.963068 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-token" (OuterVolumeSpecName: "istio-token") pod "ba071a46-6981-4c45-9b12-d9ff4d465a85" (UID: "ba071a46-6981-4c45-9b12-d9ff4d465a85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:52:18.061633 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:18.061603 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-envoy\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:52:18.061633 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:18.061629 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtpr4\" (UniqueName: \"kubernetes.io/projected/ba071a46-6981-4c45-9b12-d9ff4d465a85-kube-api-access-wtpr4\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:52:18.061633 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:18.061638 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-token\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:52:18.061633 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:18.061647 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba071a46-6981-4c45-9b12-d9ff4d465a85-istio-podinfo\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:52:18.061991 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:18.061655 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba071a46-6981-4c45-9b12-d9ff4d465a85-workload-certs\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:52:18.158472 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:18.158433 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b"] Jul 12 19:52:18.163517 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:18.163486 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ac7c507d-openshift-default-c89cf767f-7878b"] Jul 12 19:52:18.771212 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:18.771165 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:18.875687 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:18.875647 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:18.972376 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:18.972345 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:19.100598 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:19.100505 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:19.918119 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:19.918083 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba071a46-6981-4c45-9b12-d9ff4d465a85" path="/var/lib/kubelet/pods/ba071a46-6981-4c45-9b12-d9ff4d465a85/volumes" Jul 12 19:52:24.617191 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:24.617155 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:24.717173 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:24.717141 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:31.283254 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.283208 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:31.566225 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.566153 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77"] Jul 12 19:52:31.566531 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.566518 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba071a46-6981-4c45-9b12-d9ff4d465a85" containerName="istio-proxy" Jul 12 19:52:31.566584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.566534 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba071a46-6981-4c45-9b12-d9ff4d465a85" containerName="istio-proxy" Jul 12 19:52:31.566584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.566547 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba071a46-6981-4c45-9b12-d9ff4d465a85" containerName="istio-proxy" Jul 12 19:52:31.566584 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.566554 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba071a46-6981-4c45-9b12-d9ff4d465a85" containerName="istio-proxy" Jul 12 19:52:31.566689 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.566678 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba071a46-6981-4c45-9b12-d9ff4d465a85" containerName="istio-proxy" Jul 12 19:52:31.568965 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.568941 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.571572 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.571549 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-81bf5d18-openshift-default-dockercfg-r88pn\"" Jul 12 19:52:31.581391 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.581370 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77"] Jul 12 19:52:31.591914 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.591880 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:31.675995 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.675958 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-workload-certs\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.676165 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.676067 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istiod-ca-cert\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.676165 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.676130 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvnw7\" (UniqueName: \"kubernetes.io/projected/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-kube-api-access-bvnw7\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.676259 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.676164 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-data\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.676259 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.676195 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-credential-socket\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.676259 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.676229 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-podinfo\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.676259 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.676252 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-token\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.676403 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.676313 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-envoy\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.676403 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.676351 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-workload-socket\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.777124 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.777086 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-workload-certs\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.777124 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.777126 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istiod-ca-cert\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.777375 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.777170 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvnw7\" (UniqueName: \"kubernetes.io/projected/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-kube-api-access-bvnw7\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.777375 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.777210 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-data\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.777375 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.777246 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-credential-socket\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.777375 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.777270 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-podinfo\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.777375 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.777294 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-token\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.777375 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.777345 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-envoy\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.777731 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.777396 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-workload-socket\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.777802 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.777751 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-workload-certs\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.777802 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.777764 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-workload-socket\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.778197 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.778168 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-data\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.778197 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.778191 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istiod-ca-cert\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.778383 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.778226 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-credential-socket\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.779977 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.779951 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-envoy\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.780593 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.780570 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-podinfo\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.783997 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.783970 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:31.784996 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.784972 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-token\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.785153 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.785135 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvnw7\" (UniqueName: \"kubernetes.io/projected/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-kube-api-access-bvnw7\") pod \"e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:31.880894 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:31.880810 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:32.034946 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:32.034908 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77"] Jul 12 19:52:32.036873 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:52:32.036838 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa41c14b_6de7_4b6f_b4d1_213a236e55a6.slice/crio-42a382c1c8c4fc4f9b404668c8eca85592333f0df7905144716f471e581defce WatchSource:0}: Error finding container 42a382c1c8c4fc4f9b404668c8eca85592333f0df7905144716f471e581defce: Status 404 returned error can't find the container with id 42a382c1c8c4fc4f9b404668c8eca85592333f0df7905144716f471e581defce Jul 12 19:52:32.039142 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:32.039109 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:52:32.039209 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:32.039178 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:52:32.039252 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:32.039208 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:52:32.917729 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:32.917695 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" event={"ID":"aa41c14b-6de7-4b6f-b4d1-213a236e55a6","Type":"ContainerStarted","Data":"5506efd8249a223c5f6dda50807145aed9edbc68db668b90fbdd61cb35f8d4b4"} Jul 12 19:52:32.918215 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:32.917756 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" event={"ID":"aa41c14b-6de7-4b6f-b4d1-213a236e55a6","Type":"ContainerStarted","Data":"42a382c1c8c4fc4f9b404668c8eca85592333f0df7905144716f471e581defce"} Jul 12 19:52:32.937997 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:32.937949 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" podStartSLOduration=1.937933793 podStartE2EDuration="1.937933793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:52:32.936323724 +0000 UTC m=+1961.529597835" watchObservedRunningTime="2026-07-12 19:52:32.937933793 +0000 UTC m=+1961.531207890" Jul 12 19:52:33.881500 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:33.881466 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:33.886145 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:33.886120 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:33.922745 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:33.922703 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:33.923619 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:33.923601 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:34.791915 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:34.791875 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:34.866196 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:34.866159 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:34.927060 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:34.926999 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:35.193042 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:35.192981 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:35.354213 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:35.354170 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:35.455541 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:35.455448 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:39.282576 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:39.282530 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:41.108326 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:41.108259 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:41.210050 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:41.209990 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:45.985914 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:45.985874 2564 generic.go:356] "Generic (PLEG): container finished" podID="aa41c14b-6de7-4b6f-b4d1-213a236e55a6" containerID="5506efd8249a223c5f6dda50807145aed9edbc68db668b90fbdd61cb35f8d4b4" exitCode=0 Jul 12 19:52:45.986366 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:45.985946 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" event={"ID":"aa41c14b-6de7-4b6f-b4d1-213a236e55a6","Type":"ContainerDied","Data":"5506efd8249a223c5f6dda50807145aed9edbc68db668b90fbdd61cb35f8d4b4"} Jul 12 19:52:45.986415 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:45.986370 2564 scope.go:117] "RemoveContainer" containerID="5506efd8249a223c5f6dda50807145aed9edbc68db668b90fbdd61cb35f8d4b4" Jul 12 19:52:45.987580 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:45.987553 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:52:45.987685 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:45.987629 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:52:45.987685 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:45.987663 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:52:46.485194 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:46.485156 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:46.993409 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:46.993367 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" event={"ID":"aa41c14b-6de7-4b6f-b4d1-213a236e55a6","Type":"ContainerStarted","Data":"a1e95dbdf9b9d5afb15ce26ecdf640a02431cea7b218f21b8b0e89f411b1fece"} Jul 12 19:52:47.881849 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:47.881811 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:47.882948 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:47.882902 2564 patch_prober.go:28] interesting pod/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 12 19:52:47.883105 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:47.882963 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" podUID="aa41c14b-6de7-4b6f-b4d1-213a236e55a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 12 19:52:48.881806 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:48.881769 2564 patch_prober.go:28] interesting pod/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 12 19:52:48.882229 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:48.881827 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" podUID="aa41c14b-6de7-4b6f-b4d1-213a236e55a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 12 19:52:48.923041 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:48.922988 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:49.882179 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:49.882144 2564 patch_prober.go:28] interesting pod/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 12 19:52:49.882571 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:49.882233 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" podUID="aa41c14b-6de7-4b6f-b4d1-213a236e55a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 12 19:52:50.881337 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:50.881294 2564 patch_prober.go:28] interesting pod/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 12 19:52:50.881516 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:50.881351 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" podUID="aa41c14b-6de7-4b6f-b4d1-213a236e55a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 12 19:52:51.885323 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:51.885292 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:51.886411 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:51.886392 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:52:56.776575 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:56.776541 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:56.877919 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:52:56.877883 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:53:11.650344 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:53:11.650310 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:53:11.949211 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:53:11.949173 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:53:17.650525 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:53:17.650486 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:53:22.732419 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:53:22.732386 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:53:59.376242 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:53:59.376206 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:53:59.601204 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:53:59.601168 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:04.685425 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:04.685386 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:05.701413 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:05.701373 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:18.901924 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:18.901841 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:18.957717 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:18.957679 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:19.159109 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:19.159071 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77"] Jul 12 19:54:19.159371 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:19.159341 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" podUID="aa41c14b-6de7-4b6f-b4d1-213a236e55a6" containerName="istio-proxy" containerID="cri-o://a1e95dbdf9b9d5afb15ce26ecdf640a02431cea7b218f21b8b0e89f411b1fece" gracePeriod=30 Jul 12 19:54:19.179805 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:19.179767 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:20.435416 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:20.435386 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:20.530688 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:20.530654 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:20.631152 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:20.631121 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:21.319746 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:21.319711 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:21.412739 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:21.412709 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:21.514132 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:21.514080 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:21.888518 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:21.888483 2564 patch_prober.go:28] interesting pod/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 12 19:54:21.888690 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:21.888549 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" podUID="aa41c14b-6de7-4b6f-b4d1-213a236e55a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 12 19:54:22.807547 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:22.807502 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:24.440586 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.440549 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:54:24.482747 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.482702 2564 generic.go:356] "Generic (PLEG): container finished" podID="aa41c14b-6de7-4b6f-b4d1-213a236e55a6" containerID="a1e95dbdf9b9d5afb15ce26ecdf640a02431cea7b218f21b8b0e89f411b1fece" exitCode=0 Jul 12 19:54:24.482912 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.482750 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" Jul 12 19:54:24.482912 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.482757 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" event={"ID":"aa41c14b-6de7-4b6f-b4d1-213a236e55a6","Type":"ContainerDied","Data":"a1e95dbdf9b9d5afb15ce26ecdf640a02431cea7b218f21b8b0e89f411b1fece"} Jul 12 19:54:24.482912 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.482818 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77" event={"ID":"aa41c14b-6de7-4b6f-b4d1-213a236e55a6","Type":"ContainerDied","Data":"42a382c1c8c4fc4f9b404668c8eca85592333f0df7905144716f471e581defce"} Jul 12 19:54:24.482912 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.482848 2564 scope.go:117] "RemoveContainer" containerID="a1e95dbdf9b9d5afb15ce26ecdf640a02431cea7b218f21b8b0e89f411b1fece" Jul 12 19:54:24.492404 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.492377 2564 scope.go:117] "RemoveContainer" containerID="5506efd8249a223c5f6dda50807145aed9edbc68db668b90fbdd61cb35f8d4b4" Jul 12 19:54:24.500985 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.500963 2564 scope.go:117] "RemoveContainer" containerID="a1e95dbdf9b9d5afb15ce26ecdf640a02431cea7b218f21b8b0e89f411b1fece" Jul 12 19:54:24.501235 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:54:24.501218 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1e95dbdf9b9d5afb15ce26ecdf640a02431cea7b218f21b8b0e89f411b1fece\": container with ID starting with a1e95dbdf9b9d5afb15ce26ecdf640a02431cea7b218f21b8b0e89f411b1fece not found: ID does not exist" containerID="a1e95dbdf9b9d5afb15ce26ecdf640a02431cea7b218f21b8b0e89f411b1fece" Jul 12 19:54:24.501291 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.501245 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1e95dbdf9b9d5afb15ce26ecdf640a02431cea7b218f21b8b0e89f411b1fece"} err="failed to get container status \"a1e95dbdf9b9d5afb15ce26ecdf640a02431cea7b218f21b8b0e89f411b1fece\": rpc error: code = NotFound desc = could not find container \"a1e95dbdf9b9d5afb15ce26ecdf640a02431cea7b218f21b8b0e89f411b1fece\": container with ID starting with a1e95dbdf9b9d5afb15ce26ecdf640a02431cea7b218f21b8b0e89f411b1fece not found: ID does not exist" Jul 12 19:54:24.501291 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.501262 2564 scope.go:117] "RemoveContainer" containerID="5506efd8249a223c5f6dda50807145aed9edbc68db668b90fbdd61cb35f8d4b4" Jul 12 19:54:24.501526 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:54:24.501506 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5506efd8249a223c5f6dda50807145aed9edbc68db668b90fbdd61cb35f8d4b4\": container with ID starting with 5506efd8249a223c5f6dda50807145aed9edbc68db668b90fbdd61cb35f8d4b4 not found: ID does not exist" containerID="5506efd8249a223c5f6dda50807145aed9edbc68db668b90fbdd61cb35f8d4b4" Jul 12 19:54:24.501579 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.501529 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5506efd8249a223c5f6dda50807145aed9edbc68db668b90fbdd61cb35f8d4b4"} err="failed to get container status \"5506efd8249a223c5f6dda50807145aed9edbc68db668b90fbdd61cb35f8d4b4\": rpc error: code = NotFound desc = could not find container \"5506efd8249a223c5f6dda50807145aed9edbc68db668b90fbdd61cb35f8d4b4\": container with ID starting with 5506efd8249a223c5f6dda50807145aed9edbc68db668b90fbdd61cb35f8d4b4 not found: ID does not exist" Jul 12 19:54:24.566680 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.566645 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-workload-certs\") pod \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " Jul 12 19:54:24.566854 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.566684 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-podinfo\") pod \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " Jul 12 19:54:24.566854 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.566718 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvnw7\" (UniqueName: \"kubernetes.io/projected/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-kube-api-access-bvnw7\") pod \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " Jul 12 19:54:24.566854 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.566733 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-envoy\") pod \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " Jul 12 19:54:24.566854 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.566761 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istiod-ca-cert\") pod \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " Jul 12 19:54:24.566854 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.566786 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-data\") pod \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " Jul 12 19:54:24.566854 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.566802 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-credential-socket\") pod \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " Jul 12 19:54:24.566854 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.566820 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-workload-socket\") pod \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " Jul 12 19:54:24.567280 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.566880 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-token\") pod \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\" (UID: \"aa41c14b-6de7-4b6f-b4d1-213a236e55a6\") " Jul 12 19:54:24.567280 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.566953 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aa41c14b-6de7-4b6f-b4d1-213a236e55a6" (UID: "aa41c14b-6de7-4b6f-b4d1-213a236e55a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:54:24.567280 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.567215 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aa41c14b-6de7-4b6f-b4d1-213a236e55a6" (UID: "aa41c14b-6de7-4b6f-b4d1-213a236e55a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:54:24.567280 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.567242 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-workload-certs\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:54:24.567280 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.567253 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aa41c14b-6de7-4b6f-b4d1-213a236e55a6" (UID: "aa41c14b-6de7-4b6f-b4d1-213a236e55a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:54:24.567280 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.567190 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aa41c14b-6de7-4b6f-b4d1-213a236e55a6" (UID: "aa41c14b-6de7-4b6f-b4d1-213a236e55a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:54:24.567751 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.567683 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-data" (OuterVolumeSpecName: "istio-data") pod "aa41c14b-6de7-4b6f-b4d1-213a236e55a6" (UID: "aa41c14b-6de7-4b6f-b4d1-213a236e55a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:54:24.569107 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.569075 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aa41c14b-6de7-4b6f-b4d1-213a236e55a6" (UID: "aa41c14b-6de7-4b6f-b4d1-213a236e55a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 19:54:24.569223 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.569126 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aa41c14b-6de7-4b6f-b4d1-213a236e55a6" (UID: "aa41c14b-6de7-4b6f-b4d1-213a236e55a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:54:24.569685 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.569662 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-token" (OuterVolumeSpecName: "istio-token") pod "aa41c14b-6de7-4b6f-b4d1-213a236e55a6" (UID: "aa41c14b-6de7-4b6f-b4d1-213a236e55a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:54:24.569685 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.569677 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-kube-api-access-bvnw7" (OuterVolumeSpecName: "kube-api-access-bvnw7") pod "aa41c14b-6de7-4b6f-b4d1-213a236e55a6" (UID: "aa41c14b-6de7-4b6f-b4d1-213a236e55a6"). InnerVolumeSpecName "kube-api-access-bvnw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:54:24.668136 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.668104 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-token\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:54:24.668136 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.668134 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-podinfo\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:54:24.668136 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.668146 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvnw7\" (UniqueName: \"kubernetes.io/projected/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-kube-api-access-bvnw7\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:54:24.668392 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.668161 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-envoy\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:54:24.668392 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.668175 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istiod-ca-cert\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:54:24.668392 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.668188 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-istio-data\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:54:24.668392 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.668199 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-credential-socket\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:54:24.668392 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.668211 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa41c14b-6de7-4b6f-b4d1-213a236e55a6-workload-socket\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:54:24.805554 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.805515 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77"] Jul 12 19:54:24.808761 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:24.808733 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-81bf5d18-openshift-default-f8f69c99-5cb77"] Jul 12 19:54:25.917124 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:25.917084 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa41c14b-6de7-4b6f-b4d1-213a236e55a6" path="/var/lib/kubelet/pods/aa41c14b-6de7-4b6f-b4d1-213a236e55a6/volumes" Jul 12 19:54:28.515219 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:28.515180 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:28.576833 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:28.576795 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:29.157248 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:29.157193 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:34.816767 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:34.816724 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:34.881874 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:34.881828 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:34.957580 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:34.957540 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:35.061352 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:35.061321 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:45.513814 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:45.513756 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:45.837900 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:54:45.837804 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:06.770225 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:06.770190 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:12.435883 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:12.435846 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:12.493488 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:12.493453 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:12.774812 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:12.774708 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:13.049769 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.049679 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd"] Jul 12 19:55:13.050209 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.050193 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa41c14b-6de7-4b6f-b4d1-213a236e55a6" containerName="istio-proxy" Jul 12 19:55:13.050273 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.050211 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa41c14b-6de7-4b6f-b4d1-213a236e55a6" containerName="istio-proxy" Jul 12 19:55:13.050310 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.050283 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa41c14b-6de7-4b6f-b4d1-213a236e55a6" containerName="istio-proxy" Jul 12 19:55:13.050310 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.050295 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba071a46-6981-4c45-9b12-d9ff4d465a85" containerName="istio-proxy" Jul 12 19:55:13.052401 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.052384 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.054851 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.054832 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e5b064-openshift-default-dockercfg-cbjwq\"" Jul 12 19:55:13.062368 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.062343 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd"] Jul 12 19:55:13.074751 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.074714 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:13.196876 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.196839 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-token\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.196876 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.196885 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsww4\" (UniqueName: \"kubernetes.io/projected/8445869c-f001-4a39-b8e0-c29c7d3e362e-kube-api-access-hsww4\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.197180 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.196917 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8445869c-f001-4a39-b8e0-c29c7d3e362e-istiod-ca-cert\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.197180 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.196998 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-data\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.197180 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.197080 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-workload-socket\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.197180 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.197115 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-workload-certs\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.197180 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.197142 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-envoy\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.197180 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.197170 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-podinfo\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.197405 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.197252 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-credential-socket\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.298692 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.298655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-workload-certs\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.298692 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.298702 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-envoy\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.298988 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.298718 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-podinfo\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.298988 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.298770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-credential-socket\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.298988 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.298804 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-token\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.298988 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.298830 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsww4\" (UniqueName: \"kubernetes.io/projected/8445869c-f001-4a39-b8e0-c29c7d3e362e-kube-api-access-hsww4\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.298988 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.298856 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8445869c-f001-4a39-b8e0-c29c7d3e362e-istiod-ca-cert\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.298988 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.298884 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-data\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.298988 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.298938 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-workload-socket\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.299333 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.299171 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-workload-certs\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.299333 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.299276 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-credential-socket\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.299530 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.299507 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-data\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.299601 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.299538 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-workload-socket\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.299601 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.299570 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8445869c-f001-4a39-b8e0-c29c7d3e362e-istiod-ca-cert\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.301395 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.301334 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-envoy\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.301489 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.301475 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-podinfo\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.306781 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.306760 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-token\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.306905 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.306888 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsww4\" (UniqueName: \"kubernetes.io/projected/8445869c-f001-4a39-b8e0-c29c7d3e362e-kube-api-access-hsww4\") pod \"e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.363047 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.362994 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:13.383443 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.383413 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:13.507662 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.507638 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd"] Jul 12 19:55:13.509146 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:55:13.509116 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8445869c_f001_4a39_b8e0_c29c7d3e362e.slice/crio-b0cef55f319e65b58fb0475535d5c4b347886aee497d3cdaebe48a281f14e5cb WatchSource:0}: Error finding container b0cef55f319e65b58fb0475535d5c4b347886aee497d3cdaebe48a281f14e5cb: Status 404 returned error can't find the container with id b0cef55f319e65b58fb0475535d5c4b347886aee497d3cdaebe48a281f14e5cb Jul 12 19:55:13.511412 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.511381 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:55:13.511496 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.511445 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:55:13.511496 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.511472 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:55:13.735631 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.735572 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" event={"ID":"8445869c-f001-4a39-b8e0-c29c7d3e362e","Type":"ContainerStarted","Data":"9a7b185cc170f499ddaeec5a4a3b4bae38c4fb148908172077ce23d1459fd7a4"} Jul 12 19:55:13.735832 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.735652 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" event={"ID":"8445869c-f001-4a39-b8e0-c29c7d3e362e","Type":"ContainerStarted","Data":"b0cef55f319e65b58fb0475535d5c4b347886aee497d3cdaebe48a281f14e5cb"} Jul 12 19:55:13.754182 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:13.754130 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" podStartSLOduration=0.754111228 podStartE2EDuration="754.111228ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:55:13.752306314 +0000 UTC m=+2122.345580438" watchObservedRunningTime="2026-07-12 19:55:13.754111228 +0000 UTC m=+2122.347385328" Jul 12 19:55:14.363956 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:14.363918 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:14.368892 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:14.368867 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:14.741320 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:14.741280 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:14.742314 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:14.742289 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:15.623978 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:15.623938 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:15.701405 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:15.701371 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:15.806700 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:15.806655 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:15.943375 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:15.943331 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:16.268426 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:16.268336 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:16.365457 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:16.365418 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:20.113666 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:20.113637 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:20.435619 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:20.435586 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:20.727625 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:20.727532 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:21.050828 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:21.050734 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:26.483074 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:26.482998 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:26.570746 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:26.570711 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:26.636548 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:26.636505 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:26.736358 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:26.736258 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:27.041965 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:27.041871 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:27.141187 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:27.141152 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:30.783202 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:30.783161 2564 patch_prober.go:28] interesting pod/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 12 19:55:30.783589 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:30.783267 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" podUID="8445869c-f001-4a39-b8e0-c29c7d3e362e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 12 19:55:30.989355 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:30.989316 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:31.832502 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:31.832471 2564 generic.go:356] "Generic (PLEG): container finished" podID="8445869c-f001-4a39-b8e0-c29c7d3e362e" containerID="9a7b185cc170f499ddaeec5a4a3b4bae38c4fb148908172077ce23d1459fd7a4" exitCode=0 Jul 12 19:55:31.832894 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:31.832556 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" event={"ID":"8445869c-f001-4a39-b8e0-c29c7d3e362e","Type":"ContainerDied","Data":"9a7b185cc170f499ddaeec5a4a3b4bae38c4fb148908172077ce23d1459fd7a4"} Jul 12 19:55:31.832977 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:31.832952 2564 scope.go:117] "RemoveContainer" containerID="9a7b185cc170f499ddaeec5a4a3b4bae38c4fb148908172077ce23d1459fd7a4" Jul 12 19:55:31.834082 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:31.834053 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:55:31.834158 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:31.834140 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:55:31.834203 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:31.834188 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:55:32.843477 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:32.843434 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" event={"ID":"8445869c-f001-4a39-b8e0-c29c7d3e362e","Type":"ContainerStarted","Data":"92a05ecc72fd181feadad23a51654576a62d80557312b14cea812157907684b1"} Jul 12 19:55:33.363615 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:33.363576 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:33.363812 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:33.363792 2564 patch_prober.go:28] interesting pod/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 12 19:55:33.363857 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:33.363842 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" podUID="8445869c-f001-4a39-b8e0-c29c7d3e362e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 12 19:55:34.363891 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:34.363852 2564 patch_prober.go:28] interesting pod/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 12 19:55:34.364310 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:34.363910 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" podUID="8445869c-f001-4a39-b8e0-c29c7d3e362e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 12 19:55:35.364263 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:35.364220 2564 patch_prober.go:28] interesting pod/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 12 19:55:35.364662 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:35.364279 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" podUID="8445869c-f001-4a39-b8e0-c29c7d3e362e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 12 19:55:36.363549 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:36.363508 2564 patch_prober.go:28] interesting pod/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 12 19:55:36.363725 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:36.363572 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" podUID="8445869c-f001-4a39-b8e0-c29c7d3e362e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 12 19:55:36.944743 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:36.944709 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:37.043474 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:37.043438 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:37.367353 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:37.367269 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:37.367585 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:37.367561 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:37.368791 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:37.368757 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:55:47.834873 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:47.834782 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:47.932939 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:47.932904 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:49.672671 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.672631 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn"] Jul 12 19:55:49.673183 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.673162 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa41c14b-6de7-4b6f-b4d1-213a236e55a6" containerName="istio-proxy" Jul 12 19:55:49.673237 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.673187 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa41c14b-6de7-4b6f-b4d1-213a236e55a6" containerName="istio-proxy" Jul 12 19:55:49.673304 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.673293 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa41c14b-6de7-4b6f-b4d1-213a236e55a6" containerName="istio-proxy" Jul 12 19:55:49.676737 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.676718 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:49.679437 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.679407 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e5b064\"/\"default-dockercfg-vqr4j\"" Jul 12 19:55:49.680706 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.680678 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e5b064\"/\"test-model-e5b064-kserve-self-signed-certs\"" Jul 12 19:55:49.680706 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.680702 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e5b064\"/\"openshift-service-ca.crt\"" Jul 12 19:55:49.680950 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.680772 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e5b064\"/\"kube-root-ca.crt\"" Jul 12 19:55:49.684524 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.684427 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn"] Jul 12 19:55:49.697467 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.697445 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:49.715436 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.715409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-tls-certs\") pod \"test-model-e5b064-kserve-69f854d7d9-tlkbn\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:49.715436 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.715438 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-tmp-dir\") pod \"test-model-e5b064-kserve-69f854d7d9-tlkbn\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:49.715601 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.715487 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-model-cache\") pod \"test-model-e5b064-kserve-69f854d7d9-tlkbn\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:49.715645 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.715587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmt25\" (UniqueName: \"kubernetes.io/projected/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-kube-api-access-bmt25\") pod \"test-model-e5b064-kserve-69f854d7d9-tlkbn\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:49.715645 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.715636 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-dshm\") pod \"test-model-e5b064-kserve-69f854d7d9-tlkbn\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:49.715724 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.715664 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-home\") pod \"test-model-e5b064-kserve-69f854d7d9-tlkbn\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:49.816455 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.816422 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-dshm\") pod \"test-model-e5b064-kserve-69f854d7d9-tlkbn\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:49.816455 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.816454 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-home\") pod \"test-model-e5b064-kserve-69f854d7d9-tlkbn\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:49.816713 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.816481 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-tls-certs\") pod \"test-model-e5b064-kserve-69f854d7d9-tlkbn\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:49.816713 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.816496 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-tmp-dir\") pod \"test-model-e5b064-kserve-69f854d7d9-tlkbn\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:49.816713 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.816556 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-model-cache\") pod \"test-model-e5b064-kserve-69f854d7d9-tlkbn\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:49.816713 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.816606 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmt25\" (UniqueName: \"kubernetes.io/projected/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-kube-api-access-bmt25\") pod \"test-model-e5b064-kserve-69f854d7d9-tlkbn\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:49.816957 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.816918 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-home\") pod \"test-model-e5b064-kserve-69f854d7d9-tlkbn\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:49.817162 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.817137 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-tmp-dir\") pod \"test-model-e5b064-kserve-69f854d7d9-tlkbn\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:49.817234 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.817209 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-model-cache\") pod \"test-model-e5b064-kserve-69f854d7d9-tlkbn\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:49.819085 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.819058 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-dshm\") pod \"test-model-e5b064-kserve-69f854d7d9-tlkbn\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:49.819349 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.819334 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-tls-certs\") pod \"test-model-e5b064-kserve-69f854d7d9-tlkbn\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:49.824071 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.824037 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmt25\" (UniqueName: \"kubernetes.io/projected/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-kube-api-access-bmt25\") pod \"test-model-e5b064-kserve-69f854d7d9-tlkbn\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:49.987774 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:49.987691 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:50.122109 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:50.122080 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn"] Jul 12 19:55:50.124054 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:55:50.124005 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c8f61ea_61cb_4d93_8fbe_39c1a9c6c9a2.slice/crio-a6532dc6d7c159ab4a24deb85a820a296e908630687e8d3f3c5a981135418537 WatchSource:0}: Error finding container a6532dc6d7c159ab4a24deb85a820a296e908630687e8d3f3c5a981135418537: Status 404 returned error can't find the container with id a6532dc6d7c159ab4a24deb85a820a296e908630687e8d3f3c5a981135418537 Jul 12 19:55:50.937996 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:50.937940 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" event={"ID":"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2","Type":"ContainerStarted","Data":"5b0fdb499d0196dbe9188f0e4639eb6c6cb1719b9d9d348f1cc89e6236b0fb38"} Jul 12 19:55:50.937996 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:50.937986 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" event={"ID":"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2","Type":"ContainerStarted","Data":"a6532dc6d7c159ab4a24deb85a820a296e908630687e8d3f3c5a981135418537"} Jul 12 19:55:50.954066 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:50.953989 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" podStartSLOduration=1.9539724139999999 podStartE2EDuration="1.953972414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:55:50.951864748 +0000 UTC m=+2159.545138848" watchObservedRunningTime="2026-07-12 19:55:50.953972414 +0000 UTC m=+2159.547246512" Jul 12 19:55:59.987999 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:59.987971 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:55:59.988421 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:55:59.988037 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:56:00.000609 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:00.000584 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:56:01.001040 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:01.000994 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:56:16.904013 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:16.903969 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:34.666376 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:34.666345 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn"] Jul 12 19:56:34.685192 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:34.685158 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:35.563901 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:35.563870 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:35.701272 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:35.701238 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:36.105315 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:36.105272 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:36.209630 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:36.209590 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:36.310169 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:36.310141 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:36.459836 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:36.459803 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:41.183908 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:41.183861 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:49.687544 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:49.687495 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" podUID="7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2" containerName="main" containerID="cri-o://5b0fdb499d0196dbe9188f0e4639eb6c6cb1719b9d9d348f1cc89e6236b0fb38" gracePeriod=45 Jul 12 19:56:49.956526 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:49.956503 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:56:50.060027 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.059996 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-tmp-dir\") pod \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " Jul 12 19:56:50.060233 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.060081 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmt25\" (UniqueName: \"kubernetes.io/projected/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-kube-api-access-bmt25\") pod \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " Jul 12 19:56:50.060233 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.060125 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-dshm\") pod \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " Jul 12 19:56:50.060233 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.060143 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-tls-certs\") pod \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " Jul 12 19:56:50.060233 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.060158 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-home\") pod \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " Jul 12 19:56:50.060233 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.060189 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-model-cache\") pod \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\" (UID: \"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2\") " Jul 12 19:56:50.060499 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.060303 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2" (UID: "7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:50.060499 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.060446 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-home" (OuterVolumeSpecName: "home") pod "7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2" (UID: "7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:50.060610 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.060533 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2" (UID: "7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:50.060610 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.060556 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-tmp-dir\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:56:50.060610 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.060576 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-home\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:56:50.062509 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.062491 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-kube-api-access-bmt25" (OuterVolumeSpecName: "kube-api-access-bmt25") pod "7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2" (UID: "7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2"). InnerVolumeSpecName "kube-api-access-bmt25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:56:50.062765 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.062749 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2" (UID: "7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:56:50.062811 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.062773 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-dshm" (OuterVolumeSpecName: "dshm") pod "7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2" (UID: "7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:50.161869 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.161834 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmt25\" (UniqueName: \"kubernetes.io/projected/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-kube-api-access-bmt25\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:56:50.161869 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.161864 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-dshm\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:56:50.161869 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.161874 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-tls-certs\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:56:50.162130 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.161896 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2-model-cache\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:56:50.245896 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.245809 2564 generic.go:356] "Generic (PLEG): container finished" podID="7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2" containerID="5b0fdb499d0196dbe9188f0e4639eb6c6cb1719b9d9d348f1cc89e6236b0fb38" exitCode=0 Jul 12 19:56:50.245896 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.245850 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" Jul 12 19:56:50.246109 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.245895 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" event={"ID":"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2","Type":"ContainerDied","Data":"5b0fdb499d0196dbe9188f0e4639eb6c6cb1719b9d9d348f1cc89e6236b0fb38"} Jul 12 19:56:50.246109 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.245939 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn" event={"ID":"7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2","Type":"ContainerDied","Data":"a6532dc6d7c159ab4a24deb85a820a296e908630687e8d3f3c5a981135418537"} Jul 12 19:56:50.246109 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.245963 2564 scope.go:117] "RemoveContainer" containerID="5b0fdb499d0196dbe9188f0e4639eb6c6cb1719b9d9d348f1cc89e6236b0fb38" Jul 12 19:56:50.255345 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.255328 2564 scope.go:117] "RemoveContainer" containerID="5b0fdb499d0196dbe9188f0e4639eb6c6cb1719b9d9d348f1cc89e6236b0fb38" Jul 12 19:56:50.255591 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:56:50.255572 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b0fdb499d0196dbe9188f0e4639eb6c6cb1719b9d9d348f1cc89e6236b0fb38\": container with ID starting with 5b0fdb499d0196dbe9188f0e4639eb6c6cb1719b9d9d348f1cc89e6236b0fb38 not found: ID does not exist" containerID="5b0fdb499d0196dbe9188f0e4639eb6c6cb1719b9d9d348f1cc89e6236b0fb38" Jul 12 19:56:50.255637 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.255600 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b0fdb499d0196dbe9188f0e4639eb6c6cb1719b9d9d348f1cc89e6236b0fb38"} err="failed to get container status \"5b0fdb499d0196dbe9188f0e4639eb6c6cb1719b9d9d348f1cc89e6236b0fb38\": rpc error: code = NotFound desc = could not find container \"5b0fdb499d0196dbe9188f0e4639eb6c6cb1719b9d9d348f1cc89e6236b0fb38\": container with ID starting with 5b0fdb499d0196dbe9188f0e4639eb6c6cb1719b9d9d348f1cc89e6236b0fb38 not found: ID does not exist" Jul 12 19:56:50.268830 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.268806 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn"] Jul 12 19:56:50.272893 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:50.272871 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e5b064/test-model-e5b064-kserve-69f854d7d9-tlkbn"] Jul 12 19:56:51.918595 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:56:51.918561 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2" path="/var/lib/kubelet/pods/7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2/volumes" Jul 12 19:57:17.798792 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:17.798703 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:18.045388 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:18.045358 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd"] Jul 12 19:57:18.045596 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:18.045571 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" podUID="8445869c-f001-4a39-b8e0-c29c7d3e362e" containerName="istio-proxy" containerID="cri-o://92a05ecc72fd181feadad23a51654576a62d80557312b14cea812157907684b1" gracePeriod=30 Jul 12 19:57:18.070589 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:18.070495 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:18.451054 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:18.450998 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:18.846670 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:18.846551 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:18.947252 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:18.947218 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:19.119671 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:19.119592 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:22.370106 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:22.370068 2564 patch_prober.go:28] interesting pod/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 12 19:57:22.370534 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:22.370140 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" podUID="8445869c-f001-4a39-b8e0-c29c7d3e362e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 12 19:57:23.311603 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.311550 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:57:23.349246 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.349219 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-data\") pod \"8445869c-f001-4a39-b8e0-c29c7d3e362e\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " Jul 12 19:57:23.349427 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.349252 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-envoy\") pod \"8445869c-f001-4a39-b8e0-c29c7d3e362e\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " Jul 12 19:57:23.349427 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.349270 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-podinfo\") pod \"8445869c-f001-4a39-b8e0-c29c7d3e362e\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " Jul 12 19:57:23.349427 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.349306 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-token\") pod \"8445869c-f001-4a39-b8e0-c29c7d3e362e\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " Jul 12 19:57:23.349589 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.349417 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-workload-certs\") pod \"8445869c-f001-4a39-b8e0-c29c7d3e362e\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " Jul 12 19:57:23.349589 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.349479 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-workload-socket\") pod \"8445869c-f001-4a39-b8e0-c29c7d3e362e\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " Jul 12 19:57:23.349589 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.349512 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8445869c-f001-4a39-b8e0-c29c7d3e362e-istiod-ca-cert\") pod \"8445869c-f001-4a39-b8e0-c29c7d3e362e\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " Jul 12 19:57:23.349589 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.349551 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-credential-socket\") pod \"8445869c-f001-4a39-b8e0-c29c7d3e362e\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " Jul 12 19:57:23.349805 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.349591 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsww4\" (UniqueName: \"kubernetes.io/projected/8445869c-f001-4a39-b8e0-c29c7d3e362e-kube-api-access-hsww4\") pod \"8445869c-f001-4a39-b8e0-c29c7d3e362e\" (UID: \"8445869c-f001-4a39-b8e0-c29c7d3e362e\") " Jul 12 19:57:23.349887 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.349863 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8445869c-f001-4a39-b8e0-c29c7d3e362e" (UID: "8445869c-f001-4a39-b8e0-c29c7d3e362e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:57:23.349947 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.349909 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8445869c-f001-4a39-b8e0-c29c7d3e362e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8445869c-f001-4a39-b8e0-c29c7d3e362e" (UID: "8445869c-f001-4a39-b8e0-c29c7d3e362e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:57:23.350006 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.349976 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8445869c-f001-4a39-b8e0-c29c7d3e362e" (UID: "8445869c-f001-4a39-b8e0-c29c7d3e362e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:57:23.350312 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.350284 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8445869c-f001-4a39-b8e0-c29c7d3e362e" (UID: "8445869c-f001-4a39-b8e0-c29c7d3e362e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:57:23.350543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.350514 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-data" (OuterVolumeSpecName: "istio-data") pod "8445869c-f001-4a39-b8e0-c29c7d3e362e" (UID: "8445869c-f001-4a39-b8e0-c29c7d3e362e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:57:23.351735 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.351706 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8445869c-f001-4a39-b8e0-c29c7d3e362e" (UID: "8445869c-f001-4a39-b8e0-c29c7d3e362e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 19:57:23.351841 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.351817 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-token" (OuterVolumeSpecName: "istio-token") pod "8445869c-f001-4a39-b8e0-c29c7d3e362e" (UID: "8445869c-f001-4a39-b8e0-c29c7d3e362e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:57:23.351970 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.351950 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8445869c-f001-4a39-b8e0-c29c7d3e362e-kube-api-access-hsww4" (OuterVolumeSpecName: "kube-api-access-hsww4") pod "8445869c-f001-4a39-b8e0-c29c7d3e362e" (UID: "8445869c-f001-4a39-b8e0-c29c7d3e362e"). InnerVolumeSpecName "kube-api-access-hsww4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:57:23.352588 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.352562 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8445869c-f001-4a39-b8e0-c29c7d3e362e" (UID: "8445869c-f001-4a39-b8e0-c29c7d3e362e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:57:23.418614 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.418569 2564 generic.go:356] "Generic (PLEG): container finished" podID="8445869c-f001-4a39-b8e0-c29c7d3e362e" containerID="92a05ecc72fd181feadad23a51654576a62d80557312b14cea812157907684b1" exitCode=0 Jul 12 19:57:23.419132 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.418619 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" Jul 12 19:57:23.419132 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.418631 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" event={"ID":"8445869c-f001-4a39-b8e0-c29c7d3e362e","Type":"ContainerDied","Data":"92a05ecc72fd181feadad23a51654576a62d80557312b14cea812157907684b1"} Jul 12 19:57:23.419132 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.418713 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd" event={"ID":"8445869c-f001-4a39-b8e0-c29c7d3e362e","Type":"ContainerDied","Data":"b0cef55f319e65b58fb0475535d5c4b347886aee497d3cdaebe48a281f14e5cb"} Jul 12 19:57:23.419132 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.418744 2564 scope.go:117] "RemoveContainer" containerID="92a05ecc72fd181feadad23a51654576a62d80557312b14cea812157907684b1" Jul 12 19:57:23.428393 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.428373 2564 scope.go:117] "RemoveContainer" containerID="9a7b185cc170f499ddaeec5a4a3b4bae38c4fb148908172077ce23d1459fd7a4" Jul 12 19:57:23.436402 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.436380 2564 scope.go:117] "RemoveContainer" containerID="92a05ecc72fd181feadad23a51654576a62d80557312b14cea812157907684b1" Jul 12 19:57:23.436657 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:57:23.436641 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a05ecc72fd181feadad23a51654576a62d80557312b14cea812157907684b1\": container with ID starting with 92a05ecc72fd181feadad23a51654576a62d80557312b14cea812157907684b1 not found: ID does not exist" containerID="92a05ecc72fd181feadad23a51654576a62d80557312b14cea812157907684b1" Jul 12 19:57:23.436701 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.436666 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a05ecc72fd181feadad23a51654576a62d80557312b14cea812157907684b1"} err="failed to get container status \"92a05ecc72fd181feadad23a51654576a62d80557312b14cea812157907684b1\": rpc error: code = NotFound desc = could not find container \"92a05ecc72fd181feadad23a51654576a62d80557312b14cea812157907684b1\": container with ID starting with 92a05ecc72fd181feadad23a51654576a62d80557312b14cea812157907684b1 not found: ID does not exist" Jul 12 19:57:23.436701 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.436683 2564 scope.go:117] "RemoveContainer" containerID="9a7b185cc170f499ddaeec5a4a3b4bae38c4fb148908172077ce23d1459fd7a4" Jul 12 19:57:23.436916 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:57:23.436902 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a7b185cc170f499ddaeec5a4a3b4bae38c4fb148908172077ce23d1459fd7a4\": container with ID starting with 9a7b185cc170f499ddaeec5a4a3b4bae38c4fb148908172077ce23d1459fd7a4 not found: ID does not exist" containerID="9a7b185cc170f499ddaeec5a4a3b4bae38c4fb148908172077ce23d1459fd7a4" Jul 12 19:57:23.436957 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.436921 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a7b185cc170f499ddaeec5a4a3b4bae38c4fb148908172077ce23d1459fd7a4"} err="failed to get container status \"9a7b185cc170f499ddaeec5a4a3b4bae38c4fb148908172077ce23d1459fd7a4\": rpc error: code = NotFound desc = could not find container \"9a7b185cc170f499ddaeec5a4a3b4bae38c4fb148908172077ce23d1459fd7a4\": container with ID starting with 9a7b185cc170f499ddaeec5a4a3b4bae38c4fb148908172077ce23d1459fd7a4 not found: ID does not exist" Jul 12 19:57:23.439957 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.439936 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd"] Jul 12 19:57:23.445678 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.445658 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e5b064-openshift-default-6f6b57c769-8kzvd"] Jul 12 19:57:23.450234 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.450216 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-token\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:57:23.450303 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.450238 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-workload-certs\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:57:23.450303 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.450247 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-workload-socket\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:57:23.450303 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.450255 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8445869c-f001-4a39-b8e0-c29c7d3e362e-istiod-ca-cert\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:57:23.450303 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.450264 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-credential-socket\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:57:23.450303 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.450273 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsww4\" (UniqueName: \"kubernetes.io/projected/8445869c-f001-4a39-b8e0-c29c7d3e362e-kube-api-access-hsww4\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:57:23.450303 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.450285 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-data\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:57:23.450303 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.450297 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-envoy\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:57:23.450303 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.450305 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8445869c-f001-4a39-b8e0-c29c7d3e362e-istio-podinfo\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:57:23.917686 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:23.917654 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8445869c-f001-4a39-b8e0-c29c7d3e362e" path="/var/lib/kubelet/pods/8445869c-f001-4a39-b8e0-c29c7d3e362e/volumes" Jul 12 19:57:24.622666 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:24.622637 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:24.723318 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:24.723277 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:24.876512 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:24.876431 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:25.415517 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:25.415478 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:25.726345 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:25.726248 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:25.865177 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:25.865141 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:26.318589 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:26.318544 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:29.074256 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:29.074211 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:29.159396 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:29.159349 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:29.222897 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:29.222864 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:29.361505 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:29.361407 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:29.633828 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:29.633720 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:29.734890 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:29.734852 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:33.553217 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:33.553184 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:33.974473 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:33.974430 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:34.247463 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.247388 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9"] Jul 12 19:57:34.247731 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.247718 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8445869c-f001-4a39-b8e0-c29c7d3e362e" containerName="istio-proxy" Jul 12 19:57:34.247776 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.247733 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8445869c-f001-4a39-b8e0-c29c7d3e362e" containerName="istio-proxy" Jul 12 19:57:34.247776 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.247744 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8445869c-f001-4a39-b8e0-c29c7d3e362e" containerName="istio-proxy" Jul 12 19:57:34.247776 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.247750 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8445869c-f001-4a39-b8e0-c29c7d3e362e" containerName="istio-proxy" Jul 12 19:57:34.247776 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.247764 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2" containerName="main" Jul 12 19:57:34.247776 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.247770 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2" containerName="main" Jul 12 19:57:34.247941 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.247824 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8445869c-f001-4a39-b8e0-c29c7d3e362e" containerName="istio-proxy" Jul 12 19:57:34.247941 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.247831 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8445869c-f001-4a39-b8e0-c29c7d3e362e" containerName="istio-proxy" Jul 12 19:57:34.247941 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.247839 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c8f61ea-61cb-4d93-8fbe-39c1a9c6c9a2" containerName="main" Jul 12 19:57:34.250928 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.250905 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.253709 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.253690 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-5a7d9b-openshift-default-dockercfg-lrjzc\"" Jul 12 19:57:34.263639 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.263619 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9"] Jul 12 19:57:34.275556 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.275534 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:34.337970 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.337935 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-credential-socket\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.338277 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.337986 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-workload-certs\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.338277 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.338074 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-data\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.338277 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.338138 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-podinfo\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.338277 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.338169 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-token\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.338277 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.338252 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-envoy\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.338543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.338324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj5ps\" (UniqueName: \"kubernetes.io/projected/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-kube-api-access-nj5ps\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.338543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.338366 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istiod-ca-cert\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.338543 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.338417 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-workload-socket\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.438879 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.438848 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-envoy\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.439074 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.438898 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj5ps\" (UniqueName: \"kubernetes.io/projected/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-kube-api-access-nj5ps\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.439074 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.438930 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istiod-ca-cert\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.439203 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.439073 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-workload-socket\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.439203 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.439148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-credential-socket\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.439203 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.439180 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-workload-certs\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.439366 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.439212 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-data\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.439366 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.439269 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-podinfo\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.439366 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.439296 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-token\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.439755 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.439669 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-workload-socket\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.439755 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.439710 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istiod-ca-cert\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.439755 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.439729 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-data\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.439934 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.439790 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-credential-socket\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.439934 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.439919 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-workload-certs\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.441323 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.441303 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-envoy\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.441949 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.441931 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-podinfo\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.446667 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.446647 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-token\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.447072 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.447051 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj5ps\" (UniqueName: \"kubernetes.io/projected/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-kube-api-access-nj5ps\") pod \"e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.562338 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.562251 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:34.688992 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.688963 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:34.700436 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.700411 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9"] Jul 12 19:57:34.702534 ip-10-0-130-167 kubenswrapper[2564]: W0712 19:57:34.702502 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9557d6_014a_4f1d_ac16_e3a1417ccc18.slice/crio-bf4929ae4b0113815e7613accd4aee428d1d5d99e4c3ff63f8c03b9a1e15e254 WatchSource:0}: Error finding container bf4929ae4b0113815e7613accd4aee428d1d5d99e4c3ff63f8c03b9a1e15e254: Status 404 returned error can't find the container with id bf4929ae4b0113815e7613accd4aee428d1d5d99e4c3ff63f8c03b9a1e15e254 Jul 12 19:57:34.704532 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.704503 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:57:34.704633 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.704564 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:57:34.704633 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:34.704591 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:57:35.496464 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:35.496432 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" event={"ID":"8a9557d6-014a-4f1d-ac16-e3a1417ccc18","Type":"ContainerStarted","Data":"32481948753d6c9a7daa8e55ef5fc1c03d1213749d6059acecc38ae729074741"} Jul 12 19:57:35.496649 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:35.496485 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" event={"ID":"8a9557d6-014a-4f1d-ac16-e3a1417ccc18","Type":"ContainerStarted","Data":"bf4929ae4b0113815e7613accd4aee428d1d5d99e4c3ff63f8c03b9a1e15e254"} Jul 12 19:57:35.516960 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:35.516902 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" podStartSLOduration=1.5168814400000001 podStartE2EDuration="1.51688144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:57:35.514564448 +0000 UTC m=+2264.107838547" watchObservedRunningTime="2026-07-12 19:57:35.51688144 +0000 UTC m=+2264.110155539" Jul 12 19:57:35.562666 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:35.562641 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:35.567594 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:35.567570 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:36.501693 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:36.501663 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:36.502734 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:36.502714 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:37.374608 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:37.374572 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:37.434863 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:37.434829 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:37.505935 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:37.505901 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:37.641173 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:37.641064 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:37.925498 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:37.925463 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:38.025980 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:38.025944 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:41.797762 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:41.797721 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:41.898142 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:41.898080 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:42.024072 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:42.024039 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:48.571378 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:48.571342 2564 generic.go:356] "Generic (PLEG): container finished" podID="8a9557d6-014a-4f1d-ac16-e3a1417ccc18" containerID="32481948753d6c9a7daa8e55ef5fc1c03d1213749d6059acecc38ae729074741" exitCode=0 Jul 12 19:57:48.571841 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:48.571395 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" event={"ID":"8a9557d6-014a-4f1d-ac16-e3a1417ccc18","Type":"ContainerDied","Data":"32481948753d6c9a7daa8e55ef5fc1c03d1213749d6059acecc38ae729074741"} Jul 12 19:57:48.571841 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:48.571748 2564 scope.go:117] "RemoveContainer" containerID="32481948753d6c9a7daa8e55ef5fc1c03d1213749d6059acecc38ae729074741" Jul 12 19:57:48.572804 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:48.572774 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:57:48.572867 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:48.572853 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:57:48.572903 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:48.572886 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 12 19:57:49.578573 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:49.578538 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" event={"ID":"8a9557d6-014a-4f1d-ac16-e3a1417ccc18","Type":"ContainerStarted","Data":"b93e87af589d023731065da8800661775dab72aa95a94d7ab828f98ce2698a06"} Jul 12 19:57:49.850137 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:49.850044 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:50.284127 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:50.284091 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:50.545834 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:50.545743 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:50.562467 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:50.562432 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:50.563756 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:50.563728 2564 patch_prober.go:28] interesting pod/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 12 19:57:50.563868 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:50.563778 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" podUID="8a9557d6-014a-4f1d-ac16-e3a1417ccc18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 12 19:57:50.649825 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:50.649779 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:50.789180 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:50.789146 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:51.501764 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:51.501728 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:51.563697 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:51.563658 2564 patch_prober.go:28] interesting pod/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 12 19:57:51.563867 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:51.563715 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" podUID="8a9557d6-014a-4f1d-ac16-e3a1417ccc18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 12 19:57:52.563397 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:52.563358 2564 patch_prober.go:28] interesting pod/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 12 19:57:52.563813 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:52.563416 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" podUID="8a9557d6-014a-4f1d-ac16-e3a1417ccc18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 12 19:57:53.626924 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:53.626881 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:53.628366 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:53.628331 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:57:56.108759 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:56.108727 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:56.173646 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:56.173615 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:56.361737 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:56.361649 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:56.711704 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:56.711671 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:57.099566 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:57.099476 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:57.201109 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:57.201074 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:57.301504 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:57.301460 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:57.371699 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:57:57.371606 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:02.923080 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:02.923040 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:02.987489 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:02.987449 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:03.151129 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:03.151097 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9"] Jul 12 19:58:03.151377 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:03.151355 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" podUID="8a9557d6-014a-4f1d-ac16-e3a1417ccc18" containerName="istio-proxy" containerID="cri-o://b93e87af589d023731065da8800661775dab72aa95a94d7ab828f98ce2698a06" gracePeriod=30 Jul 12 19:58:03.175671 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:03.175571 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:03.694845 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:03.694804 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:04.000002 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:03.999914 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:04.507514 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:04.507483 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:07.267089 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:07.267048 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:07.336445 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:07.336400 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:07.400362 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:07.400328 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:07.758415 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:07.758380 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:07.933212 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:07.933180 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:08.034244 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.034160 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:08.407817 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.407790 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:58:08.471847 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.471813 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-workload-certs\") pod \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " Jul 12 19:58:08.471847 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.471849 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-workload-socket\") pod \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " Jul 12 19:58:08.472165 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.471883 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-token\") pod \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " Jul 12 19:58:08.472165 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.471910 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-envoy\") pod \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " Jul 12 19:58:08.472165 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.471960 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj5ps\" (UniqueName: \"kubernetes.io/projected/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-kube-api-access-nj5ps\") pod \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " Jul 12 19:58:08.472165 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.471993 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istiod-ca-cert\") pod \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " Jul 12 19:58:08.472165 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.472076 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-podinfo\") pod \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " Jul 12 19:58:08.472165 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.472113 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8a9557d6-014a-4f1d-ac16-e3a1417ccc18" (UID: "8a9557d6-014a-4f1d-ac16-e3a1417ccc18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:58:08.472165 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.472157 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8a9557d6-014a-4f1d-ac16-e3a1417ccc18" (UID: "8a9557d6-014a-4f1d-ac16-e3a1417ccc18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:58:08.472533 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.472206 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-credential-socket\") pod \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " Jul 12 19:58:08.472533 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.472240 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-data\") pod \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\" (UID: \"8a9557d6-014a-4f1d-ac16-e3a1417ccc18\") " Jul 12 19:58:08.472533 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.472457 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8a9557d6-014a-4f1d-ac16-e3a1417ccc18" (UID: "8a9557d6-014a-4f1d-ac16-e3a1417ccc18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:58:08.472533 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.472512 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8a9557d6-014a-4f1d-ac16-e3a1417ccc18" (UID: "8a9557d6-014a-4f1d-ac16-e3a1417ccc18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:58:08.472750 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.472575 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-credential-socket\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:58:08.472750 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.472595 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-workload-certs\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:58:08.472750 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.472608 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-workload-socket\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:58:08.473545 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.473519 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-data" (OuterVolumeSpecName: "istio-data") pod "8a9557d6-014a-4f1d-ac16-e3a1417ccc18" (UID: "8a9557d6-014a-4f1d-ac16-e3a1417ccc18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:58:08.474474 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.474429 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8a9557d6-014a-4f1d-ac16-e3a1417ccc18" (UID: "8a9557d6-014a-4f1d-ac16-e3a1417ccc18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 19:58:08.474474 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.474430 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8a9557d6-014a-4f1d-ac16-e3a1417ccc18" (UID: "8a9557d6-014a-4f1d-ac16-e3a1417ccc18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:58:08.474625 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.474530 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-token" (OuterVolumeSpecName: "istio-token") pod "8a9557d6-014a-4f1d-ac16-e3a1417ccc18" (UID: "8a9557d6-014a-4f1d-ac16-e3a1417ccc18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:58:08.474625 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.474540 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-kube-api-access-nj5ps" (OuterVolumeSpecName: "kube-api-access-nj5ps") pod "8a9557d6-014a-4f1d-ac16-e3a1417ccc18" (UID: "8a9557d6-014a-4f1d-ac16-e3a1417ccc18"). InnerVolumeSpecName "kube-api-access-nj5ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:58:08.573402 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.573362 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-token\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:58:08.573402 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.573394 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-envoy\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:58:08.573402 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.573403 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj5ps\" (UniqueName: \"kubernetes.io/projected/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-kube-api-access-nj5ps\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:58:08.573402 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.573412 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istiod-ca-cert\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:58:08.573666 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.573421 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-podinfo\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:58:08.573666 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.573428 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9557d6-014a-4f1d-ac16-e3a1417ccc18-istio-data\") on node \"ip-10-0-130-167.ec2.internal\" DevicePath \"\"" Jul 12 19:58:08.677434 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.677404 2564 generic.go:356] "Generic (PLEG): container finished" podID="8a9557d6-014a-4f1d-ac16-e3a1417ccc18" containerID="b93e87af589d023731065da8800661775dab72aa95a94d7ab828f98ce2698a06" exitCode=0 Jul 12 19:58:08.677611 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.677454 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" Jul 12 19:58:08.677611 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.677488 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" event={"ID":"8a9557d6-014a-4f1d-ac16-e3a1417ccc18","Type":"ContainerDied","Data":"b93e87af589d023731065da8800661775dab72aa95a94d7ab828f98ce2698a06"} Jul 12 19:58:08.677611 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.677548 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9" event={"ID":"8a9557d6-014a-4f1d-ac16-e3a1417ccc18","Type":"ContainerDied","Data":"bf4929ae4b0113815e7613accd4aee428d1d5d99e4c3ff63f8c03b9a1e15e254"} Jul 12 19:58:08.677611 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.677587 2564 scope.go:117] "RemoveContainer" containerID="b93e87af589d023731065da8800661775dab72aa95a94d7ab828f98ce2698a06" Jul 12 19:58:08.687801 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.687783 2564 scope.go:117] "RemoveContainer" containerID="32481948753d6c9a7daa8e55ef5fc1c03d1213749d6059acecc38ae729074741" Jul 12 19:58:08.696042 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.696010 2564 scope.go:117] "RemoveContainer" containerID="b93e87af589d023731065da8800661775dab72aa95a94d7ab828f98ce2698a06" Jul 12 19:58:08.696507 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:58:08.696474 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b93e87af589d023731065da8800661775dab72aa95a94d7ab828f98ce2698a06\": container with ID starting with b93e87af589d023731065da8800661775dab72aa95a94d7ab828f98ce2698a06 not found: ID does not exist" containerID="b93e87af589d023731065da8800661775dab72aa95a94d7ab828f98ce2698a06" Jul 12 19:58:08.696616 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.696516 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b93e87af589d023731065da8800661775dab72aa95a94d7ab828f98ce2698a06"} err="failed to get container status \"b93e87af589d023731065da8800661775dab72aa95a94d7ab828f98ce2698a06\": rpc error: code = NotFound desc = could not find container \"b93e87af589d023731065da8800661775dab72aa95a94d7ab828f98ce2698a06\": container with ID starting with b93e87af589d023731065da8800661775dab72aa95a94d7ab828f98ce2698a06 not found: ID does not exist" Jul 12 19:58:08.696616 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.696535 2564 scope.go:117] "RemoveContainer" containerID="32481948753d6c9a7daa8e55ef5fc1c03d1213749d6059acecc38ae729074741" Jul 12 19:58:08.696879 ip-10-0-130-167 kubenswrapper[2564]: E0712 19:58:08.696810 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32481948753d6c9a7daa8e55ef5fc1c03d1213749d6059acecc38ae729074741\": container with ID starting with 32481948753d6c9a7daa8e55ef5fc1c03d1213749d6059acecc38ae729074741 not found: ID does not exist" containerID="32481948753d6c9a7daa8e55ef5fc1c03d1213749d6059acecc38ae729074741" Jul 12 19:58:08.696879 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.696842 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32481948753d6c9a7daa8e55ef5fc1c03d1213749d6059acecc38ae729074741"} err="failed to get container status \"32481948753d6c9a7daa8e55ef5fc1c03d1213749d6059acecc38ae729074741\": rpc error: code = NotFound desc = could not find container \"32481948753d6c9a7daa8e55ef5fc1c03d1213749d6059acecc38ae729074741\": container with ID starting with 32481948753d6c9a7daa8e55ef5fc1c03d1213749d6059acecc38ae729074741 not found: ID does not exist" Jul 12 19:58:08.699411 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.699389 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9"] Jul 12 19:58:08.702871 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:08.702848 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5a7d9b-openshift-default-b668c68fc-kmwd9"] Jul 12 19:58:09.916736 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:09.916703 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a9557d6-014a-4f1d-ac16-e3a1417ccc18" path="/var/lib/kubelet/pods/8a9557d6-014a-4f1d-ac16-e3a1417ccc18/volumes" Jul 12 19:58:11.745451 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:11.745400 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:12.058986 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:12.058886 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:12.361416 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:12.361325 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:12.598836 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:12.598801 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:15.457277 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:15.457228 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:15.522236 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:15.522190 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:15.585064 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:15.585011 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:15.772052 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:15.771938 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:16.117189 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:16.117105 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:16.217608 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:16.217575 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:19.841138 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:19.841075 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:24.240683 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:24.240643 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:24.344104 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:24.344074 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:28.630134 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:28.630092 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:28.730463 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:28.730428 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:37.727719 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:37.727679 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:37.828006 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:58:37.827968 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 19:59:20.694078 ip-10-0-130-167 kubenswrapper[2564]: I0712 19:59:20.694039 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:00:44.975375 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:00:44.975341 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:00:45.076324 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:00:45.076269 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:24.615363 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:24.615319 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:25.018103 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:25.018068 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:25.122493 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:25.122457 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:25.265835 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:25.265801 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:30.966129 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:30.966088 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:31.033770 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:31.033735 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:31.229487 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:31.229408 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:31.672576 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:31.672541 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:31.988418 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:31.988317 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:32.102891 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:32.102859 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:32.203273 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:32.203238 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:32.333878 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:32.333801 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:37.523206 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:37.523166 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:37.623743 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:37.623708 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:37.766799 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:37.766763 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:38.544379 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:38.544351 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68b74f5cfd-jcnhj_89f785c3-9ed2-49fa-9cae-9447fa673963/authorino/0.log" Jul 12 20:01:42.862643 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:42.862558 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ghffv_8b8b400e-ccf0-4a14-b275-fa3bed8078ef/manager/0.log" Jul 12 20:01:42.975071 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:42.975039 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-956xx_8df0c6a3-1593-4291-9e35-52e4b6a85a53/manager/0.log" Jul 12 20:01:43.213960 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:43.213927 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zkrk2_a5f65c6d-a6bf-4977-a40e-704613ac348b/server/0.log" Jul 12 20:01:43.331993 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:43.331956 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-bm6ks_6133c191-d1e4-4476-b2c8-aeee1b3d7c4e/manager/0.log" Jul 12 20:01:43.683263 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:43.683235 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68dff8c9d6-vgj7g_393fd51b-4d9e-4f83-9eac-82d93b1d7d69/copy-manifests/0.log" Jul 12 20:01:43.701372 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:43.701341 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68dff8c9d6-vgj7g_393fd51b-4d9e-4f83-9eac-82d93b1d7d69/manager/0.log" Jul 12 20:01:44.449858 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:44.449827 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs_e1af07a3-642b-4014-8dc0-bdec0818d9b5/util/0.log" Jul 12 20:01:44.455615 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:44.455592 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs_e1af07a3-642b-4014-8dc0-bdec0818d9b5/pull/0.log" Jul 12 20:01:44.461297 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:44.461264 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs_e1af07a3-642b-4014-8dc0-bdec0818d9b5/extract/0.log" Jul 12 20:01:44.571091 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:44.571061 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc_673bd84f-85af-496f-b902-c74c065574fa/util/0.log" Jul 12 20:01:44.577166 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:44.577145 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc_673bd84f-85af-496f-b902-c74c065574fa/pull/0.log" Jul 12 20:01:44.583045 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:44.583007 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc_673bd84f-85af-496f-b902-c74c065574fa/extract/0.log" Jul 12 20:01:44.688665 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:44.688636 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl_a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67/util/0.log" Jul 12 20:01:44.694266 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:44.694245 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl_a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67/pull/0.log" Jul 12 20:01:44.699660 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:44.699640 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl_a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67/extract/0.log" Jul 12 20:01:44.804218 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:44.804140 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb_3e8eb60c-253a-4201-b30c-561a435b44ab/util/0.log" Jul 12 20:01:44.810118 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:44.810093 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb_3e8eb60c-253a-4201-b30c-561a435b44ab/pull/0.log" Jul 12 20:01:44.815766 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:44.815742 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb_3e8eb60c-253a-4201-b30c-561a435b44ab/extract/0.log" Jul 12 20:01:44.958413 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:44.958385 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68b74f5cfd-jcnhj_89f785c3-9ed2-49fa-9cae-9447fa673963/authorino/0.log" Jul 12 20:01:45.071657 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:45.071567 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-6zcqm_7bae773d-e2a2-4e14-86c1-3822c2223fd5/manager/0.log" Jul 12 20:01:45.280957 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:45.280921 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-pmvq9_ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9/kuadrant-console-plugin/0.log" Jul 12 20:01:45.397172 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:45.397094 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-f9lgb_6e13e221-b298-49fa-9392-f3ce487b978b/registry-server/0.log" Jul 12 20:01:45.517053 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:45.517000 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-ftr4m_1497e0f8-a76f-40a3-927f-c772158234d9/manager/0.log" Jul 12 20:01:45.741809 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:45.741778 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-kxblk_be6cf8a0-8822-41f2-a079-905b10525b94/manager/0.log" Jul 12 20:01:46.070934 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:46.070843 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4x6vqt_916b483e-49cb-4db0-91d8-1c4d9b4020cd/istio-proxy/0.log" Jul 12 20:01:46.402263 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:46.402170 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59ff7fb449-z4k9f_3d20bc6a-a256-4490-951e-e38d805189b7/kube-auth-proxy/0.log" Jul 12 20:01:46.523812 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:46.523780 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lb5c2_371d7bb4-bbea-4dac-8af4-67bcce7bc561/istio-proxy/1.log" Jul 12 20:01:46.748347 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:46.748316 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-2hh49_c071e130-cd90-4008-8e90-25090615fb17/payload-processing/0.log" Jul 12 20:01:47.407058 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:47.407005 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-r44np_c602a26e-93f6-4960-9d58-4fc728139e8a/main/0.log" Jul 12 20:01:47.752257 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:47.752181 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-f48rc_6b4fad4c-ed4c-4470-9f13-663f6de6d7da/main/0.log" Jul 12 20:01:52.171907 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.171866 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-txwrt/must-gather-z9zmz"] Jul 12 20:01:52.172302 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.172199 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9557d6-014a-4f1d-ac16-e3a1417ccc18" containerName="istio-proxy" Jul 12 20:01:52.172302 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.172210 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9557d6-014a-4f1d-ac16-e3a1417ccc18" containerName="istio-proxy" Jul 12 20:01:52.172302 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.172280 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a9557d6-014a-4f1d-ac16-e3a1417ccc18" containerName="istio-proxy" Jul 12 20:01:52.172403 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.172348 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9557d6-014a-4f1d-ac16-e3a1417ccc18" containerName="istio-proxy" Jul 12 20:01:52.172403 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.172354 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9557d6-014a-4f1d-ac16-e3a1417ccc18" containerName="istio-proxy" Jul 12 20:01:52.172473 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.172411 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a9557d6-014a-4f1d-ac16-e3a1417ccc18" containerName="istio-proxy" Jul 12 20:01:52.175900 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.175878 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-txwrt/must-gather-z9zmz" Jul 12 20:01:52.178352 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.178321 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-txwrt\"/\"openshift-service-ca.crt\"" Jul 12 20:01:52.178478 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.178356 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-txwrt\"/\"kube-root-ca.crt\"" Jul 12 20:01:52.179544 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.179525 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-txwrt\"/\"default-dockercfg-44w77\"" Jul 12 20:01:52.183678 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.183655 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-txwrt/must-gather-z9zmz"] Jul 12 20:01:52.185494 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.185470 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:52.246171 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.246138 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:52.262247 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.262218 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-665lk\" (UniqueName: \"kubernetes.io/projected/6ff84e70-a578-47ad-b88c-d94117027a28-kube-api-access-665lk\") pod \"must-gather-z9zmz\" (UID: \"6ff84e70-a578-47ad-b88c-d94117027a28\") " pod="openshift-must-gather-txwrt/must-gather-z9zmz" Jul 12 20:01:52.262409 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.262317 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ff84e70-a578-47ad-b88c-d94117027a28-must-gather-output\") pod \"must-gather-z9zmz\" (UID: \"6ff84e70-a578-47ad-b88c-d94117027a28\") " pod="openshift-must-gather-txwrt/must-gather-z9zmz" Jul 12 20:01:52.362868 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.362835 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ff84e70-a578-47ad-b88c-d94117027a28-must-gather-output\") pod \"must-gather-z9zmz\" (UID: \"6ff84e70-a578-47ad-b88c-d94117027a28\") " pod="openshift-must-gather-txwrt/must-gather-z9zmz" Jul 12 20:01:52.363065 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.362879 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-665lk\" (UniqueName: \"kubernetes.io/projected/6ff84e70-a578-47ad-b88c-d94117027a28-kube-api-access-665lk\") pod \"must-gather-z9zmz\" (UID: \"6ff84e70-a578-47ad-b88c-d94117027a28\") " pod="openshift-must-gather-txwrt/must-gather-z9zmz" Jul 12 20:01:52.363216 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.363197 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ff84e70-a578-47ad-b88c-d94117027a28-must-gather-output\") pod \"must-gather-z9zmz\" (UID: \"6ff84e70-a578-47ad-b88c-d94117027a28\") " pod="openshift-must-gather-txwrt/must-gather-z9zmz" Jul 12 20:01:52.377466 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.377446 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-665lk\" (UniqueName: \"kubernetes.io/projected/6ff84e70-a578-47ad-b88c-d94117027a28-kube-api-access-665lk\") pod \"must-gather-z9zmz\" (UID: \"6ff84e70-a578-47ad-b88c-d94117027a28\") " pod="openshift-must-gather-txwrt/must-gather-z9zmz" Jul 12 20:01:52.485620 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.485514 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-txwrt/must-gather-z9zmz" Jul 12 20:01:52.622647 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.622623 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-txwrt/must-gather-z9zmz"] Jul 12 20:01:52.624244 ip-10-0-130-167 kubenswrapper[2564]: W0712 20:01:52.624219 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ff84e70_a578_47ad_b88c_d94117027a28.slice/crio-d5829a209fbd662b629bedd5fdfcee32d0522c9d7456cdfacd26a02ae3bed08a WatchSource:0}: Error finding container d5829a209fbd662b629bedd5fdfcee32d0522c9d7456cdfacd26a02ae3bed08a: Status 404 returned error can't find the container with id d5829a209fbd662b629bedd5fdfcee32d0522c9d7456cdfacd26a02ae3bed08a Jul 12 20:01:52.626005 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.625990 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 20:01:52.766953 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:52.766861 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-txwrt/must-gather-z9zmz" event={"ID":"6ff84e70-a578-47ad-b88c-d94117027a28","Type":"ContainerStarted","Data":"d5829a209fbd662b629bedd5fdfcee32d0522c9d7456cdfacd26a02ae3bed08a"} Jul 12 20:01:53.778652 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:53.778607 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-txwrt/must-gather-z9zmz" event={"ID":"6ff84e70-a578-47ad-b88c-d94117027a28","Type":"ContainerStarted","Data":"80f0e2a0d59e6b3486b82da38875f05e1b3972855eb2639a15172143df3614a2"} Jul 12 20:01:53.779197 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:53.778669 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-txwrt/must-gather-z9zmz" event={"ID":"6ff84e70-a578-47ad-b88c-d94117027a28","Type":"ContainerStarted","Data":"d48867bee771fcbe75c6ca898880bd8383396e50d64e24b17baf2e553b836cf7"} Jul 12 20:01:53.795915 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:53.794213 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-txwrt/must-gather-z9zmz" podStartSLOduration=1.056967655 podStartE2EDuration="1.794189684s" totalImagesPullingTime="737.222029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 20:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 20:01:53.793101799 +0000 UTC m=+2522.386375900" watchObservedRunningTime="2026-07-12 20:01:53.794189684 +0000 UTC m=+2522.387463783" Jul 12 20:01:54.921867 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:54.921830 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nd2b6_697ae350-ebe5-4859-a7d4-389fd1da28f8/global-pull-secret-syncer/0.log" Jul 12 20:01:55.010109 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:55.010076 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" Jul 12 20:01:55.109238 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:55.109207 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-167.ec2.internal_c9f6b081b0c599a6d0bb3dfcb98d7c97/haproxy/0.log" Jul 12 20:01:59.569230 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:59.569197 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs_e1af07a3-642b-4014-8dc0-bdec0818d9b5/extract/0.log" Jul 12 20:01:59.595791 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:59.595743 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs_e1af07a3-642b-4014-8dc0-bdec0818d9b5/util/0.log" Jul 12 20:01:59.619979 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:59.619915 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759w2rqs_e1af07a3-642b-4014-8dc0-bdec0818d9b5/pull/0.log" Jul 12 20:01:59.653574 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:59.653540 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc_673bd84f-85af-496f-b902-c74c065574fa/extract/0.log" Jul 12 20:01:59.671195 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:59.671139 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc_673bd84f-85af-496f-b902-c74c065574fa/util/0.log" Jul 12 20:01:59.695344 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:59.695312 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bgfbc_673bd84f-85af-496f-b902-c74c065574fa/pull/0.log" Jul 12 20:01:59.729585 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:59.729404 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl_a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67/extract/0.log" Jul 12 20:01:59.748036 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:59.747984 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl_a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67/util/0.log" Jul 12 20:01:59.768224 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:59.768194 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsknl_a9fefec5-6d6d-4837-a7e4-f9fb93b6ce67/pull/0.log" Jul 12 20:01:59.797773 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:59.797745 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb_3e8eb60c-253a-4201-b30c-561a435b44ab/extract/0.log" Jul 12 20:01:59.829499 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:59.829406 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb_3e8eb60c-253a-4201-b30c-561a435b44ab/util/0.log" Jul 12 20:01:59.853701 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:01:59.853666 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vwvjb_3e8eb60c-253a-4201-b30c-561a435b44ab/pull/0.log" Jul 12 20:02:00.267188 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:00.267148 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68b74f5cfd-jcnhj_89f785c3-9ed2-49fa-9cae-9447fa673963/authorino/0.log" Jul 12 20:02:00.296001 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:00.295971 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-6zcqm_7bae773d-e2a2-4e14-86c1-3822c2223fd5/manager/0.log" Jul 12 20:02:00.359260 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:00.359232 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-pmvq9_ce2f6ec1-e4cf-4a36-a39d-662a3c970ed9/kuadrant-console-plugin/0.log" Jul 12 20:02:00.403962 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:00.403932 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-f9lgb_6e13e221-b298-49fa-9392-f3ce487b978b/registry-server/0.log" Jul 12 20:02:00.574184 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:00.573804 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-ftr4m_1497e0f8-a76f-40a3-927f-c772158234d9/manager/0.log" Jul 12 20:02:00.690278 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:00.690241 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-kxblk_be6cf8a0-8822-41f2-a079-905b10525b94/manager/0.log" Jul 12 20:02:02.476764 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:02.476729 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c8q7b_0da2150f-4939-40fa-b34e-80957d3b4416/node-exporter/0.log" Jul 12 20:02:02.498230 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:02.498196 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c8q7b_0da2150f-4939-40fa-b34e-80957d3b4416/kube-rbac-proxy/0.log" Jul 12 20:02:02.520511 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:02.520481 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c8q7b_0da2150f-4939-40fa-b34e-80957d3b4416/init-textfile/0.log" Jul 12 20:02:04.440258 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:04.440225 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-kvdtw_5efb57d2-5bf0-4799-8e22-3a3066c94cb2/networking-console-plugin/0.log" Jul 12 20:02:05.365557 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.365518 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-txwrt/perf-node-gather-daemonset-64286"] Jul 12 20:02:05.373590 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.373565 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" Jul 12 20:02:05.377991 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.377906 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-txwrt/perf-node-gather-daemonset-64286"] Jul 12 20:02:05.494595 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.494554 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/433ec0ee-b23a-401b-9719-6aa0f77c238a-lib-modules\") pod \"perf-node-gather-daemonset-64286\" (UID: \"433ec0ee-b23a-401b-9719-6aa0f77c238a\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" Jul 12 20:02:05.495252 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.495223 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/433ec0ee-b23a-401b-9719-6aa0f77c238a-sys\") pod \"perf-node-gather-daemonset-64286\" (UID: \"433ec0ee-b23a-401b-9719-6aa0f77c238a\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" Jul 12 20:02:05.495413 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.495395 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/433ec0ee-b23a-401b-9719-6aa0f77c238a-proc\") pod \"perf-node-gather-daemonset-64286\" (UID: \"433ec0ee-b23a-401b-9719-6aa0f77c238a\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" Jul 12 20:02:05.495531 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.495517 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/433ec0ee-b23a-401b-9719-6aa0f77c238a-podres\") pod \"perf-node-gather-daemonset-64286\" (UID: \"433ec0ee-b23a-401b-9719-6aa0f77c238a\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" Jul 12 20:02:05.495707 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.495690 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glq5v\" (UniqueName: \"kubernetes.io/projected/433ec0ee-b23a-401b-9719-6aa0f77c238a-kube-api-access-glq5v\") pod \"perf-node-gather-daemonset-64286\" (UID: \"433ec0ee-b23a-401b-9719-6aa0f77c238a\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" Jul 12 20:02:05.596410 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.596379 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/433ec0ee-b23a-401b-9719-6aa0f77c238a-podres\") pod \"perf-node-gather-daemonset-64286\" (UID: \"433ec0ee-b23a-401b-9719-6aa0f77c238a\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" Jul 12 20:02:05.596617 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.596470 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glq5v\" (UniqueName: \"kubernetes.io/projected/433ec0ee-b23a-401b-9719-6aa0f77c238a-kube-api-access-glq5v\") pod \"perf-node-gather-daemonset-64286\" (UID: \"433ec0ee-b23a-401b-9719-6aa0f77c238a\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" Jul 12 20:02:05.596617 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.596518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/433ec0ee-b23a-401b-9719-6aa0f77c238a-lib-modules\") pod \"perf-node-gather-daemonset-64286\" (UID: \"433ec0ee-b23a-401b-9719-6aa0f77c238a\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" Jul 12 20:02:05.596617 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.596571 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/433ec0ee-b23a-401b-9719-6aa0f77c238a-podres\") pod \"perf-node-gather-daemonset-64286\" (UID: \"433ec0ee-b23a-401b-9719-6aa0f77c238a\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" Jul 12 20:02:05.596617 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.596576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/433ec0ee-b23a-401b-9719-6aa0f77c238a-sys\") pod \"perf-node-gather-daemonset-64286\" (UID: \"433ec0ee-b23a-401b-9719-6aa0f77c238a\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" Jul 12 20:02:05.596796 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.596632 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/433ec0ee-b23a-401b-9719-6aa0f77c238a-sys\") pod \"perf-node-gather-daemonset-64286\" (UID: \"433ec0ee-b23a-401b-9719-6aa0f77c238a\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" Jul 12 20:02:05.596796 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.596643 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/433ec0ee-b23a-401b-9719-6aa0f77c238a-proc\") pod \"perf-node-gather-daemonset-64286\" (UID: \"433ec0ee-b23a-401b-9719-6aa0f77c238a\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" Jul 12 20:02:05.596796 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.596679 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/433ec0ee-b23a-401b-9719-6aa0f77c238a-lib-modules\") pod \"perf-node-gather-daemonset-64286\" (UID: \"433ec0ee-b23a-401b-9719-6aa0f77c238a\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" Jul 12 20:02:05.596796 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.596730 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/433ec0ee-b23a-401b-9719-6aa0f77c238a-proc\") pod \"perf-node-gather-daemonset-64286\" (UID: \"433ec0ee-b23a-401b-9719-6aa0f77c238a\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" Jul 12 20:02:05.603873 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.603847 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glq5v\" (UniqueName: \"kubernetes.io/projected/433ec0ee-b23a-401b-9719-6aa0f77c238a-kube-api-access-glq5v\") pod \"perf-node-gather-daemonset-64286\" (UID: \"433ec0ee-b23a-401b-9719-6aa0f77c238a\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" Jul 12 20:02:05.687617 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.687583 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" Jul 12 20:02:05.851331 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.851242 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-txwrt/perf-node-gather-daemonset-64286"] Jul 12 20:02:05.856962 ip-10-0-130-167 kubenswrapper[2564]: W0712 20:02:05.856934 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod433ec0ee_b23a_401b_9719_6aa0f77c238a.slice/crio-f7126076b41afac1c000ce5d50c18b6a967ff8f386f9c96610785ff6633c1c5f WatchSource:0}: Error finding container f7126076b41afac1c000ce5d50c18b6a967ff8f386f9c96610785ff6633c1c5f: Status 404 returned error can't find the container with id f7126076b41afac1c000ce5d50c18b6a967ff8f386f9c96610785ff6633c1c5f Jul 12 20:02:05.867887 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:05.867863 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" event={"ID":"433ec0ee-b23a-401b-9719-6aa0f77c238a","Type":"ContainerStarted","Data":"f7126076b41afac1c000ce5d50c18b6a967ff8f386f9c96610785ff6633c1c5f"} Jul 12 20:02:06.875290 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:06.875245 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" event={"ID":"433ec0ee-b23a-401b-9719-6aa0f77c238a","Type":"ContainerStarted","Data":"89e5175705292ed8e0decefe4bb551405baae9c64bcb1188bd09c3abbe29ddd5"} Jul 12 20:02:06.875683 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:06.875329 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" Jul 12 20:02:06.890794 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:06.890749 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" podStartSLOduration=1.890733899 podStartE2EDuration="1.890733899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 20:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 20:02:06.889049216 +0000 UTC m=+2535.482323312" watchObservedRunningTime="2026-07-12 20:02:06.890733899 +0000 UTC m=+2535.484007997" Jul 12 20:02:07.231995 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:07.231963 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zvbnn_174905d8-e6cd-4e9b-a6fa-02d6f96d05d5/dns/0.log" Jul 12 20:02:07.250808 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:07.250785 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zvbnn_174905d8-e6cd-4e9b-a6fa-02d6f96d05d5/kube-rbac-proxy/0.log" Jul 12 20:02:07.272546 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:07.272510 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-grbbl_9a01d1fa-13e1-47af-b3af-2a9de7431dd1/dns-node-resolver/0.log" Jul 12 20:02:07.911685 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:07.911656 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-thbcj_ab20a71c-5305-4984-8c11-e3b5e5451fcb/node-ca/0.log" Jul 12 20:02:08.797347 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:08.797314 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4x6vqt_916b483e-49cb-4db0-91d8-1c4d9b4020cd/istio-proxy/0.log" Jul 12 20:02:08.946716 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:08.946680 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59ff7fb449-z4k9f_3d20bc6a-a256-4490-951e-e38d805189b7/kube-auth-proxy/0.log" Jul 12 20:02:08.975736 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:08.975705 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lb5c2_371d7bb4-bbea-4dac-8af4-67bcce7bc561/istio-proxy/0.log" Jul 12 20:02:09.036341 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:09.036305 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lb5c2_371d7bb4-bbea-4dac-8af4-67bcce7bc561/istio-proxy/1.log" Jul 12 20:02:09.147843 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:09.147744 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-2hh49_c071e130-cd90-4008-8e90-25090615fb17/payload-processing/0.log" Jul 12 20:02:09.824341 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:09.824311 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9c99k_b6b3ac65-ccd5-4d7b-8809-8372731b54be/serve-healthcheck-canary/0.log" Jul 12 20:02:10.379564 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:10.379533 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dtmkq_46d9ad51-4ee6-44b6-9416-5f161034d19f/kube-rbac-proxy/0.log" Jul 12 20:02:10.399421 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:10.399384 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dtmkq_46d9ad51-4ee6-44b6-9416-5f161034d19f/exporter/0.log" Jul 12 20:02:10.419698 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:10.419662 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dtmkq_46d9ad51-4ee6-44b6-9416-5f161034d19f/extractor/0.log" Jul 12 20:02:12.510892 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:12.510858 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ghffv_8b8b400e-ccf0-4a14-b275-fa3bed8078ef/manager/0.log" Jul 12 20:02:12.709624 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:12.709589 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-956xx_8df0c6a3-1593-4291-9e35-52e4b6a85a53/manager/0.log" Jul 12 20:02:12.806195 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:12.806117 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zkrk2_a5f65c6d-a6bf-4977-a40e-704613ac348b/server/0.log" Jul 12 20:02:12.894045 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:12.893988 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-64286" Jul 12 20:02:12.904947 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:12.904908 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-bm6ks_6133c191-d1e4-4476-b2c8-aeee1b3d7c4e/manager/0.log" Jul 12 20:02:13.166859 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:13.166821 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68dff8c9d6-vgj7g_393fd51b-4d9e-4f83-9eac-82d93b1d7d69/manager/0.log" Jul 12 20:02:13.182693 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:13.182661 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68dff8c9d6-vgj7g_393fd51b-4d9e-4f83-9eac-82d93b1d7d69/copy-manifests/0.log" Jul 12 20:02:14.513619 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:14.513589 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-gkh4l_d39fc2e8-d1dd-4f6f-99f8-7fd4c1b66b81/openshift-lws-operator/0.log" Jul 12 20:02:15.627661 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:15.627583 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4x6vqt_916b483e-49cb-4db0-91d8-1c4d9b4020cd/istio-proxy/0.log" Jul 12 20:02:15.750771 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:15.750735 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59ff7fb449-z4k9f_3d20bc6a-a256-4490-951e-e38d805189b7/kube-auth-proxy/0.log" Jul 12 20:02:15.773184 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:15.773147 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lb5c2_371d7bb4-bbea-4dac-8af4-67bcce7bc561/istio-proxy/0.log" Jul 12 20:02:15.828731 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:15.828696 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lb5c2_371d7bb4-bbea-4dac-8af4-67bcce7bc561/istio-proxy/1.log" Jul 12 20:02:15.922923 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:15.922894 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-2hh49_c071e130-cd90-4008-8e90-25090615fb17/payload-processing/0.log" Jul 12 20:02:17.201976 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:17.201944 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4x6vqt_916b483e-49cb-4db0-91d8-1c4d9b4020cd/istio-proxy/0.log" Jul 12 20:02:17.328315 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:17.328280 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59ff7fb449-z4k9f_3d20bc6a-a256-4490-951e-e38d805189b7/kube-auth-proxy/0.log" Jul 12 20:02:17.351148 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:17.351091 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lb5c2_371d7bb4-bbea-4dac-8af4-67bcce7bc561/istio-proxy/0.log" Jul 12 20:02:17.408872 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:17.408810 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lb5c2_371d7bb4-bbea-4dac-8af4-67bcce7bc561/istio-proxy/1.log" Jul 12 20:02:17.505630 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:17.505518 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-2hh49_c071e130-cd90-4008-8e90-25090615fb17/payload-processing/0.log" Jul 12 20:02:21.112656 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:21.112629 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xv5j2_39bf2054-7b11-434a-a6e3-4d1f04cdd1a3/kube-multus-additional-cni-plugins/0.log" Jul 12 20:02:21.133885 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:21.133860 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xv5j2_39bf2054-7b11-434a-a6e3-4d1f04cdd1a3/egress-router-binary-copy/0.log" Jul 12 20:02:21.152928 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:21.152898 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xv5j2_39bf2054-7b11-434a-a6e3-4d1f04cdd1a3/cni-plugins/0.log" Jul 12 20:02:21.170480 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:21.170456 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xv5j2_39bf2054-7b11-434a-a6e3-4d1f04cdd1a3/bond-cni-plugin/0.log" Jul 12 20:02:21.188996 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:21.188972 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xv5j2_39bf2054-7b11-434a-a6e3-4d1f04cdd1a3/routeoverride-cni/0.log" Jul 12 20:02:21.207277 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:21.207235 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xv5j2_39bf2054-7b11-434a-a6e3-4d1f04cdd1a3/whereabouts-cni-bincopy/0.log" Jul 12 20:02:21.226370 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:21.226333 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xv5j2_39bf2054-7b11-434a-a6e3-4d1f04cdd1a3/whereabouts-cni/0.log" Jul 12 20:02:21.267923 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:21.267887 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fdm9d_ea178531-de7c-44c8-911b-308836d78022/kube-multus/0.log" Jul 12 20:02:21.389468 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:21.389379 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fz99s_8a0fd497-acc1-4391-bb37-3a00c2c3362d/network-metrics-daemon/0.log" Jul 12 20:02:21.405888 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:21.405858 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fz99s_8a0fd497-acc1-4391-bb37-3a00c2c3362d/kube-rbac-proxy/0.log" Jul 12 20:02:21.415451 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:21.415421 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4vrt_a7a4e516-ce2a-4b83-8d99-83c61331242f/konnectivity-agent/0.log" limit=1024 Jul 12 20:02:22.457871 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:22.457837 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jj2hs_0c1cc216-5111-43d9-979f-ad6cc1beece6/ovn-controller/0.log" Jul 12 20:02:22.473129 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:22.473100 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jj2hs_0c1cc216-5111-43d9-979f-ad6cc1beece6/ovn-acl-logging/0.log" Jul 12 20:02:22.501327 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:22.501294 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jj2hs_0c1cc216-5111-43d9-979f-ad6cc1beece6/kube-rbac-proxy-node/0.log" Jul 12 20:02:22.526530 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:22.526502 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jj2hs_0c1cc216-5111-43d9-979f-ad6cc1beece6/kube-rbac-proxy-ovn-metrics/0.log" Jul 12 20:02:22.543093 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:22.543067 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jj2hs_0c1cc216-5111-43d9-979f-ad6cc1beece6/northd/0.log" Jul 12 20:02:22.562934 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:22.562895 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jj2hs_0c1cc216-5111-43d9-979f-ad6cc1beece6/nbdb/0.log" Jul 12 20:02:22.581833 ip-10-0-130-167 kubenswrapper[2564]: I0712 20:02:22.581808 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jj2hs_0c1cc216-5111-43d9-979f-ad6cc1beece6/sbdb/0.log"